Server Admin Log/Archive 89

== 2025-01-31 ==
* 23:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1218 (T384592)', diff saved to https://phabricator.wikimedia.org/P72970 and previous config saved to /var/cache/conftool/dbconfig/20250131-234543-marostegui.json
* 23:45 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db1218.eqiad.wmnet with reason: Maintenance
* 23:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T384592)', diff saved to https://phabricator.wikimedia.org/P72969 and previous config saved to /var/cache/conftool/dbconfig/20250131-234521-marostegui.json
* 23:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P72968 and previous config saved to /var/cache/conftool/dbconfig/20250131-233014-marostegui.json
* 23:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P72967 and previous config saved to /var/cache/conftool/dbconfig/20250131-231507-marostegui.json
* 23:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T384592)', diff saved to https://phabricator.wikimedia.org/P72966 and previous config saved to /var/cache/conftool/dbconfig/20250131-230000-marostegui.json
* 21:38 mforns@deploy2002: Finished deploy [airflow-dags/analytics@16e825c]: update commons impact metrics category allow list (duration: 01m 10s)
* 21:37 mforns@deploy2002: Started deploy [airflow-dags/analytics@16e825c]: update commons impact metrics category allow list
* 21:29 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1207 (T384592)', diff saved to https://phabricator.wikimedia.org/P72964 and previous config saved to /var/cache/conftool/dbconfig/20250131-212934-marostegui.json
* 21:29 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db1207.eqiad.wmnet with reason: Maintenance
* 21:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T384592)', diff saved to https://phabricator.wikimedia.org/P72963 and previous config saved to /var/cache/conftool/dbconfig/20250131-212912-marostegui.json
* 21:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P72962 and previous config saved to /var/cache/conftool/dbconfig/20250131-211406-marostegui.json
* 20:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P72961 and previous config saved to /var/cache/conftool/dbconfig/20250131-205859-marostegui.json
* 20:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T384592)', diff saved to https://phabricator.wikimedia.org/P72960 and previous config saved to /var/cache/conftool/dbconfig/20250131-204353-marostegui.json
* 19:59 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ms-be2088.codfw.wmnet with reason: T381919
* 19:23 aokoth@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 19:23 aokoth@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 19:06 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1206 (T384592)', diff saved to https://phabricator.wikimedia.org/P72959 and previous config saved to /var/cache/conftool/dbconfig/20250131-190627-marostegui.json
* 19:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db1206.eqiad.wmnet with reason: Maintenance
* 19:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T384592)', diff saved to https://phabricator.wikimedia.org/P72958 and previous config saved to /var/cache/conftool/dbconfig/20250131-190605-marostegui.json
* 18:57 marostegui@cumin1002: dbctl commit (dc=all): 'db1233 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72957 and previous config saved to /var/cache/conftool/dbconfig/20250131-185722-root.json
* 18:57 marostegui@cumin1002: dbctl commit (dc=all): 'db2225 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72956 and previous config saved to /var/cache/conftool/dbconfig/20250131-185715-root.json
* 18:50 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P72955 and previous config saved to /var/cache/conftool/dbconfig/20250131-185058-marostegui.json
* 18:42 marostegui@cumin1002: dbctl commit (dc=all): 'db1233 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72954 and previous config saved to /var/cache/conftool/dbconfig/20250131-184217-root.json
* 18:42 marostegui@cumin1002: dbctl commit (dc=all): 'db2225 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72953 and previous config saved to /var/cache/conftool/dbconfig/20250131-184210-root.json
* 18:35 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P72952 and previous config saved to /var/cache/conftool/dbconfig/20250131-183551-marostegui.json
* 18:28 dwisehaupt@dns1004: END - running authdns-update
* 18:27 marostegui@cumin1002: dbctl commit (dc=all): 'db1233 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72951 and previous config saved to /var/cache/conftool/dbconfig/20250131-182712-root.json
* 18:27 marostegui@cumin1002: dbctl commit (dc=all): 'db2225 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72950 and previous config saved to /var/cache/conftool/dbconfig/20250131-182705-root.json
* 18:26 dwisehaupt@dns1004: START - running authdns-update
* 18:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T384592)', diff saved to https://phabricator.wikimedia.org/P72949 and previous config saved to /var/cache/conftool/dbconfig/20250131-182044-marostegui.json
* 18:12 marostegui@cumin1002: dbctl commit (dc=all): 'db1233 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72948 and previous config saved to /var/cache/conftool/dbconfig/20250131-181206-root.json
* 18:12 marostegui@cumin1002: dbctl commit (dc=all): 'db2225 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72947 and previous config saved to /var/cache/conftool/dbconfig/20250131-181200-root.json
* 17:59 jhathaway@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on sretest1002.eqiad.wmnet with reason: T381919
* 17:57 marostegui@cumin1002: dbctl commit (dc=all): 'db1233 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72946 and previous config saved to /var/cache/conftool/dbconfig/20250131-175701-root.json
* 17:56 marostegui@cumin1002: dbctl commit (dc=all): 'db2225 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72945 and previous config saved to /var/cache/conftool/dbconfig/20250131-175654-root.json
* 17:52 jhathaway@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on sretest1001.eqiad.wmnet with reason: T381919
* 17:25 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mr1-magru,mr1-magru IPv6,mr1-magru.oob,mr1-magru.oob IPv6 with reason: emergency reboot
* 17:10 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1196 (T384592)', diff saved to https://phabricator.wikimedia.org/P72943 and previous config saved to /var/cache/conftool/dbconfig/20250131-171027-marostegui.json
* 17:10 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 17:10 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db1196.eqiad.wmnet with reason: Maintenance
* 17:09 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1195 (T384592)', diff saved to https://phabricator.wikimedia.org/P72942 and previous config saved to /var/cache/conftool/dbconfig/20250131-170947-marostegui.json
* 16:54 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P72941 and previous config saved to /var/cache/conftool/dbconfig/20250131-165440-marostegui.json
* 16:39 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P72940 and previous config saved to /var/cache/conftool/dbconfig/20250131-163933-marostegui.json
* 16:24 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1195 (T384592)', diff saved to https://phabricator.wikimedia.org/P72939 and previous config saved to /var/cache/conftool/dbconfig/20250131-162426-marostegui.json
* 15:30 fnegri@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1013.eqiad.wmnet,service=s1
* 15:30 fnegri@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1013.eqiad.wmnet,service=s3
* 15:27 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72936 and previous config saved to /var/cache/conftool/dbconfig/20250131-152726-root.json
* 15:23 fnegri@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddb1013.eqiad.wmnet
* 15:21 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host puppetserver2004
* 15:21 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host puppetserver2004
* 15:20 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db2243
* 15:20 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host db2243
* 15:20 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1195 (T384592)', diff saved to https://phabricator.wikimedia.org/P72935 and previous config saved to /var/cache/conftool/dbconfig/20250131-152008-marostegui.json
* 15:20 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db1195.eqiad.wmnet with reason: Maintenance
* 15:19 fnegri@cumin1002: START - Cookbook sre.hosts.reboot-single for host clouddb1013.eqiad.wmnet
* 15:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T384592)', diff saved to https://phabricator.wikimedia.org/P72934 and previous config saved to /var/cache/conftool/dbconfig/20250131-151947-marostegui.json
* 15:19 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:18 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:18 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:16 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:16 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding db2243 to codfw - jhancock@cumin2002"
* 15:16 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding db2243 to codfw - jhancock@cumin2002"
* 15:15 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:13 fnegri@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1013.eqiad.wmnet with reason: Rebooting clouddb1013 T384946
* 15:12 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:12 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72933 and previous config saved to /var/cache/conftool/dbconfig/20250131-151221-root.json
* 15:12 fnegri@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1013.eqiad.wmnet,service=s3
* 15:11 fnegri@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1013.eqiad.wmnet,service=s1
* 15:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P72931 and previous config saved to /var/cache/conftool/dbconfig/20250131-150441-marostegui.json
* 14:57 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72930 and previous config saved to /var/cache/conftool/dbconfig/20250131-145715-root.json
* 14:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P72929 and previous config saved to /var/cache/conftool/dbconfig/20250131-144934-marostegui.json
* 14:48 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1233.eqiad.wmnet with reason: Index rebuild
* 14:47 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2225.codfw.wmnet with reason: Index rebuild
* 14:47 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2225.codfw.wmnet
* 14:47 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1233.eqiad.wmnet
* 14:42 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72928 and previous config saved to /var/cache/conftool/dbconfig/20250131-144210-root.json
* 14:41 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2225.codfw.wmnet
* 14:41 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1233.eqiad.wmnet
* 14:41 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1233 db2225 T385084', diff saved to https://phabricator.wikimedia.org/P72927 and previous config saved to /var/cache/conftool/dbconfig/20250131-144103-marostegui.json
* 14:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T384592)', diff saved to https://phabricator.wikimedia.org/P72926 and previous config saved to /var/cache/conftool/dbconfig/20250131-143428-marostegui.json
* 14:29 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 14:28 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 14:27 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 14:27 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72925 and previous config saved to /var/cache/conftool/dbconfig/20250131-142705-root.json
* 14:26 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 14:21 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 14:20 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 14:19 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 14:19 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 14:19 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 14:19 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 14:18 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 14:17 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 14:16 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 14:16 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 14:11 marostegui@cumin1002: dbctl commit (dc=all): 'db2238 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72924 and previous config saved to /var/cache/conftool/dbconfig/20250131-141153-root.json
* 14:11 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72923 and previous config saved to /var/cache/conftool/dbconfig/20250131-141143-root.json
* 14:10 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 14:06 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 14:06 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 14:05 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 14:04 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 14:04 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 14:04 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 14:04 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 13:58 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 13:56 marostegui@cumin1002: dbctl commit (dc=all): 'db2238 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72922 and previous config saved to /var/cache/conftool/dbconfig/20250131-135648-root.json
* 13:56 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72921 and previous config saved to /var/cache/conftool/dbconfig/20250131-135637-root.json
* 13:56 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 13:54 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 13:53 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 13:53 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 13:43 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 13:42 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 13:41 marostegui@cumin1002: dbctl commit (dc=all): 'db2238 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72920 and previous config saved to /var/cache/conftool/dbconfig/20250131-134142-root.json
* 13:41 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 13:41 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72919 and previous config saved to /var/cache/conftool/dbconfig/20250131-134132-root.json
* 13:41 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 13:40 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 13:39 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 13:38 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 13:32 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 13:32 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 13:31 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 13:28 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 13:28 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 13:27 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 13:27 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 13:26 marostegui@cumin1002: dbctl commit (dc=all): 'db2238 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72918 and previous config saved to /var/cache/conftool/dbconfig/20250131-132637-root.json
* 13:26 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72917 and previous config saved to /var/cache/conftool/dbconfig/20250131-132626-root.json
* 13:23 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1186 (T384592)', diff saved to https://phabricator.wikimedia.org/P72916 and previous config saved to /var/cache/conftool/dbconfig/20250131-132335-marostegui.json
* 13:23 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db1186.eqiad.wmnet with reason: Maintenance
* 13:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T384592)', diff saved to https://phabricator.wikimedia.org/P72915 and previous config saved to /var/cache/conftool/dbconfig/20250131-132313-marostegui.json
* 13:20 jelto@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Test GitLab 17.7 on Replica
* 13:12 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Test GitLab 17.7 on Replica
* 13:11 marostegui@cumin1002: dbctl commit (dc=all): 'db2238 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72914 and previous config saved to /var/cache/conftool/dbconfig/20250131-131132-root.json
* 13:11 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72913 and previous config saved to /var/cache/conftool/dbconfig/20250131-131121-root.json
* 13:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P72912 and previous config saved to /var/cache/conftool/dbconfig/20250131-130806-marostegui.json
* 13:05 urbanecm: Restart mw-debug.codfw.pinkunicorn (verifying fix for T384919)
* 13:02 jynus: ran apt clean on grafana2001
* 12:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P72911 and previous config saved to /var/cache/conftool/dbconfig/20250131-125258-marostegui.json
* 12:38 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1115833ForeignApi: Do not share AJAX options between requests (T385261) (duration: 09m 49s)
* 12:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T384592)', diff saved to https://phabricator.wikimedia.org/P72910 and previous config saved to /var/cache/conftool/dbconfig/20250131-123751-marostegui.json
* 12:32 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde: Continuing with sync
* 12:31 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde: Backport for gerrit:1115833ForeignApi: Do not share AJAX options between requests (T385261) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 12:28 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1115833ForeignApi: Do not share AJAX options between requests (T385261)
* 12:26 jayme@cumin1002: END (FAIL) - Cookbook sre.k8s.wipe-cluster (exit_code=99) Wipe the K8s cluster staging-codfw: Kubernetes upgrade
* 12:16 moritzm: rebalance codfw/D following OS updates T382508
* 12:10 jayme@cumin1002: START - Cookbook sre.k8s.wipe-cluster Wipe the K8s cluster staging-codfw: Kubernetes upgrade
* 12:08 jayme@cumin1002: END (FAIL) - Cookbook sre.k8s.wipe-cluster (exit_code=99) Wipe the K8s cluster staging-codfw: Kubernetes upgrade
* 12:04 jayme@cumin1002: START - Cookbook sre.k8s.wipe-cluster Wipe the K8s cluster staging-codfw: Kubernetes upgrade
* 12:03 jayme@cumin1002: END (FAIL) - Cookbook sre.k8s.wipe-cluster (exit_code=99) Wipe the K8s cluster staging-codfw: Kubernetes upgrade
* 11:59 jayme@cumin1002: START - Cookbook sre.k8s.wipe-cluster Wipe the K8s cluster staging-codfw: Kubernetes upgrade
* 11:40 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1189.eqiad.wmnet with reason: Index rebuild
* 11:38 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1189.eqiad.wmnet
* 11:37 jayme@cumin1002: END (FAIL) - Cookbook sre.k8s.wipe-cluster (exit_code=99) Wipe the K8s cluster staging-codfw: Kubernetes upgrade
* 11:36 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on dbstore1008.eqiad.wmnet with reason: maintenance
* 11:33 marostegui: Upgrade mariadb in dbstore1008 and rebuild tables on s1, s5 and s7 T384818
* 11:33 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1169 (T384592)', diff saved to https://phabricator.wikimedia.org/P72908 and previous config saved to /var/cache/conftool/dbconfig/20250131-113321-marostegui.json
* 11:33 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db1169.eqiad.wmnet with reason: Maintenance
* 11:33 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1008.eqiad.wmnet with reason: maintenance
* 11:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T384592)', diff saved to https://phabricator.wikimedia.org/P72907 and previous config saved to /var/cache/conftool/dbconfig/20250131-113300-marostegui.json
* 11:31 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1189.eqiad.wmnet
* 11:30 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1189 T385051', diff saved to https://phabricator.wikimedia.org/P72906 and previous config saved to /var/cache/conftool/dbconfig/20250131-113011-marostegui.json
* 11:29 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1223 to s3 primary T385051', diff saved to https://phabricator.wikimedia.org/P72905 and previous config saved to /var/cache/conftool/dbconfig/20250131-112920-root.json
* 11:29 marostegui: Starting s3 eqiad failover from db1189 to db1223 - T385051
* 11:27 jayme@cumin1002: START - Cookbook sre.k8s.wipe-cluster Wipe the K8s cluster staging-codfw: Kubernetes upgrade
* 11:26 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s3 T385051
* 11:26 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1223 with weight 0 T385051', diff saved to https://phabricator.wikimedia.org/P72904 and previous config saved to /var/cache/conftool/dbconfig/20250131-112614-root.json
* 11:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P72903 and previous config saved to /var/cache/conftool/dbconfig/20250131-111753-marostegui.json
* 11:08 jayme@cumin1002: END (FAIL) - Cookbook sre.k8s.wipe-cluster (exit_code=99) Wipe the K8s cluster staging-codfw: Kubernetes upgrade
* 11:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P72901 and previous config saved to /var/cache/conftool/dbconfig/20250131-110246-marostegui.json
* 11:02 kevinbazira@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' .
* 10:53 akosiaris@deploy2002: helmfile [staging] DONE helmfile.d/services/mw-api-int: apply
* 10:53 akosiaris@deploy2002: helmfile [staging] START helmfile.d/services/mw-api-int: apply
* 10:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T384592)', diff saved to https://phabricator.wikimedia.org/P72900 and previous config saved to /var/cache/conftool/dbconfig/20250131-104739-marostegui.json
* 10:16 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2139.codfw.wmnet
* 10:16 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:16 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2139.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002"
* 10:15 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2139.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002"
* 10:14 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2238.codfw.wmnet with reason: Index rebuild
* 10:13 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2238.codfw.wmnet
* 10:13 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1246.eqiad.wmnet with reason: Index rebuild
* 10:12 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1246.eqiad.wmnet
* 10:08 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2238.codfw.wmnet
* 10:08 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2238', diff saved to https://phabricator.wikimedia.org/P72899 and previous config saved to /var/cache/conftool/dbconfig/20250131-100806-marostegui.json
* 10:06 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1246.eqiad.wmnet
* 10:06 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1246', diff saved to https://phabricator.wikimedia.org/P72898 and previous config saved to /var/cache/conftool/dbconfig/20250131-100650-marostegui.json
* 10:03 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 09:41 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1163 (T384592)', diff saved to https://phabricator.wikimedia.org/P72897 and previous config saved to /var/cache/conftool/dbconfig/20250131-094152-marostegui.json
* 09:41 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db1163.eqiad.wmnet with reason: Maintenance
* 09:40 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2139.codfw.wmnet
* 09:37 slyngshede@dns1004: END - running authdns-update
* 09:35 slyngshede@dns1004: START - running authdns-update
* 09:17 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 09:16 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 09:06 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 09:05 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 08:52 moritzm: rebalance codfw/A following OS updates T382508
* 08:37 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2029.codfw.wmnet to cluster codfw and group A
* 08:37 kartik@deploy2002: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' .
* 08:34 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2029.codfw.wmnet to cluster codfw and group A
* 08:31 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2029.codfw.wmnet
* 08:22 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2029.codfw.wmnet
* 08:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2029.codfw.wmnet with OS bookworm
* 07:56 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2029.codfw.wmnet with reason: host reimage
* 07:52 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2029.codfw.wmnet with reason: host reimage
* 07:30 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2029.codfw.wmnet with OS bookworm
* 06:10 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2209.codfw.wmnet with reason: Maintenance
* 06:10 marostegui: Deploy schema change on s5 codfw with replication enabled dbmaint T385183
* 05:47 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1189.eqiad.wmnet with reason: Maintenance
* 03:22 zabe@deploy2002: Finished scap sync-world: Backport for gerrit:1115582Update interwiki cache, gerrit:1115590configure kncwiki (T385185) (duration: 10m 28s)
* 03:15 zabe@deploy2002: zabe: Continuing with sync
* 03:14 zabe@deploy2002: zabe: Backport for gerrit:1115582Update interwiki cache, gerrit:1115590configure kncwiki (T385185) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 03:11 zabe@deploy2002: Started scap sync-world: Backport for gerrit:1115582Update interwiki cache, gerrit:1115590configure kncwiki (T385185)
* 02:49 zabe@deploy2002: Finished scap sync-world: Backport for gerrit:1115577Activate kncwiki (T385181) (duration: 11m 02s)
* 02:42 zabe@deploy2002: zabe: Continuing with sync
* 02:40 zabe@deploy2002: zabe: Backport for gerrit:1115577Activate kncwiki (T385181) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 02:37 zabe@deploy2002: Started scap sync-world: Backport for gerrit:1115577Activate kncwiki (T385181)
* 02:32 zabe@deploy2002: Finished scap sync-world: Backport for gerrit:1115567Prepare kncwiki (T385181) (duration: 09m 10s)
* 02:26 zabe@deploy2002: zabe: Continuing with sync
* 02:25 zabe@deploy2002: zabe: Backport for gerrit:1115567Prepare kncwiki (T385181) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 02:23 zabe@deploy2002: Started scap sync-world: Backport for gerrit:1115567Prepare kncwiki (T385181)
* 01:12 pt1979@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1254.eqiad.wmnet with OS bookworm
* 01:12 pt1979@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin1002"
* 01:11 pt1979@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin1002"
* 01:02 zabe@deploy2002: Finished scap sync-world: Backport for gerrit:1115542hooks: Stop using deprecated DatabaseBlock method, gerrit:1115146Restrict editing on ruwikimedia (T382805) (duration: 11m 08s)
* 00:56 zabe@deploy2002: zabe: Continuing with sync
* 00:54 zabe@deploy2002: zabe: Backport for gerrit:1115542hooks: Stop using deprecated DatabaseBlock method, gerrit:1115146Restrict editing on ruwikimedia (T382805) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 00:53 pt1979@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1254.eqiad.wmnet with reason: host reimage
* 00:51 zabe@deploy2002: Started scap sync-world: Backport for gerrit:1115542hooks: Stop using deprecated DatabaseBlock method, gerrit:1115146Restrict editing on ruwikimedia (T382805)
* 00:50 pt1979@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1254.eqiad.wmnet with reason: host reimage
* 00:43 pt1979@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1253.eqiad.wmnet with OS bookworm
* 00:42 pt1979@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin1002"
* 00:42 pt1979@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin1002"
* 00:33 pt1979@cumin1002: START - Cookbook sre.hosts.reimage for host db1254.eqiad.wmnet with OS bookworm
* 00:30 pt1979@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1254.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 00:25 pt1979@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1253.eqiad.wmnet with reason: host reimage
* 00:21 pt1979@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1253.eqiad.wmnet with reason: host reimage
* 00:07 pt1979@cumin1002: START - Cookbook sre.hosts.provision for host db1254.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 00:06 pt1979@cumin1002: START - Cookbook sre.hosts.reimage for host db1253.eqiad.wmnet with OS bookworm
* 00:05 pt1979@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1252.eqiad.wmnet with OS bookworm
* 00:05 pt1979@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin1002"
* 00:04 pt1979@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin1002"
== 2025-01-30 ==
* 23:57 urbanecm@deploy2002: Finished scap sync-world: Backport for gerrit:1115404CommunityConfiguration: Enable on all wikis except locked down (T383910) (duration: 13m 41s)
* 23:51 urbanecm@deploy2002: urbanecm: Continuing with sync
* 23:51 pt1979@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1253.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 23:47 pt1979@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1252.eqiad.wmnet with reason: host reimage
* 23:46 urbanecm@deploy2002: urbanecm: Backport for gerrit:1115404CommunityConfiguration: Enable on all wikis except locked down (T383910) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 23:44 urbanecm@deploy2002: Started scap sync-world: Backport for gerrit:1115404CommunityConfiguration: Enable on all wikis except locked down (T383910)
* 23:43 pt1979@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1252.eqiad.wmnet with reason: host reimage
* 23:43 pt1979@cumin1002: START - Cookbook sre.hosts.provision for host db1253.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 23:41 urbanecm@deploy2002: Finished scap sync-world: Backport for gerrit:1115403Babel: Enable CommunityConfiguration on all wikis (T374348) (duration: 13m 04s)
* 23:41 pt1979@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1253.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 23:34 urbanecm@deploy2002: urbanecm: Continuing with sync
* 23:30 urbanecm@deploy2002: urbanecm: Backport for gerrit:1115403Babel: Enable CommunityConfiguration on all wikis (T374348) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 23:28 pt1979@cumin1002: START - Cookbook sre.hosts.reimage for host db1252.eqiad.wmnet with OS bookworm
* 23:27 urbanecm@deploy2002: Started scap sync-world: Backport for gerrit:1115403Babel: Enable CommunityConfiguration on all wikis (T374348)
* 23:25 pt1979@cumin1002: START - Cookbook sre.hosts.provision for host db1253.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 23:25 pt1979@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1253.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 23:23 urbanecm@deploy2002: Finished scap sync-world: Backport for [[gerrit:1114002[testwiki] Babel: Enable CommunityConfiguration integration (T374348)]] (duration: 15m 32s)
* 23:17 urbanecm@deploy2002: urbanecm: Continuing with sync
* 23:10 urbanecm@deploy2002: urbanecm: Backport for [[gerrit:1114002[testwiki] Babel: Enable CommunityConfiguration integration (T374348)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 23:07 urbanecm@deploy2002: Started scap sync-world: Backport for [[gerrit:1114002[testwiki] Babel: Enable CommunityConfiguration integration (T374348)]]
* 22:33 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on netflow2003.codfw.wmnet with reason: disabling gnmic in systemd
* 22:31 pt1979@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1252.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:57 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ms-be2088.codfw.wmnet with reason: T381919
* 21:56 cjming: end of UTC late backport window
* 21:51 cjming@deploy2002: Finished scap sync-world: Backport for gerrit:1115492Update and sync 404 error handler pages (T383952) (duration: 10m 59s)
* 21:45 cjming@deploy2002: cjming, matmarex: Continuing with sync
* 21:43 cjming@deploy2002: cjming, matmarex: Backport for gerrit:1115492Update and sync 404 error handler pages (T383952) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:40 cjming@deploy2002: Started scap sync-world: Backport for gerrit:1115492Update and sync 404 error handler pages (T383952)
* 21:32 pt1979@cumin1002: START - Cookbook sre.hosts.provision for host db1253.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:23 tgr@deploy2002: Finished scap sync-world: Backport for gerrit:1115369Do not disable extensions on SUL3 shared authentication domain (T373737 T384919 T384236) (duration: 12m 42s)
* 21:17 tgr@deploy2002: tgr: Continuing with sync
* 21:13 tgr@deploy2002: tgr: Backport for gerrit:1115369Do not disable extensions on SUL3 shared authentication domain (T373737 T384919 T384236) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:10 tgr@deploy2002: Started scap sync-world: Backport for gerrit:1115369Do not disable extensions on SUL3 shared authentication domain (T373737 T384919 T384236)
* 21:09 pt1979@cumin1002: START - Cookbook sre.hosts.provision for host db1252.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:06 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 21:06 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add dns entries for db1252,db1253,db1254 - pt1979@cumin2002"
* 21:06 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add dns entries for db1252,db1253,db1254 - pt1979@cumin2002"
* 21:04 cdanis@deploy2002: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
* 21:04 cstone: payments-wiki upgraded from to 
* 21:03 cdanis@deploy2002: helmfile [staging] START helmfile.d/services/chart-renderer: apply
* 21:01 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 20:47 cdanis@deploy2002: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
* 20:47 cdanis@deploy2002: helmfile [staging] START helmfile.d/services/chart-renderer: apply
* 20:29 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 20:27 jhathaway@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ms-be2088.codfw.wmnet with reason: T381919
* 20:26 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 20:24 denisse: Backing up Grafana DB on grafana1002 - T384840
* 19:44 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2239.codfw.wmnet with reason: Maintenance
* 19:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2227 (T384592)', diff saved to https://phabricator.wikimedia.org/P72894 and previous config saved to /var/cache/conftool/dbconfig/20250130-194354-marostegui.json
* 19:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P72893 and previous config saved to /var/cache/conftool/dbconfig/20250130-192847-marostegui.json
* 19:14 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 19:13 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P72892 and previous config saved to /var/cache/conftool/dbconfig/20250130-191340-marostegui.json
* 19:12 jhuneidi@deploy2002: rebuilt and synchronized wikiversions files: group2 to 1.44.0-wmf.14 refs T382365
* 19:11 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 18:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2227 (T384592)', diff saved to https://phabricator.wikimedia.org/P72891 and previous config saved to /var/cache/conftool/dbconfig/20250130-185833-marostegui.json
* 18:48 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add dns entries for frdb1007,fran1002 and frban1002 - pt1979@cumin2002"
* 18:44 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 18:10 bd808@deploy2002: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 18:10 bd808@deploy2002: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 18:10 bd808@deploy2002: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 18:09 bd808@deploy2002: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 18:08 bd808@deploy2002: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 18:07 bd808@deploy2002: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 18:06 ladsgroup@deploy2002: Finished scap sync-world: Backport for gerrit:1115466file: Remove from filerevision when only one row exists (T384481) (duration: 10m 48s)
* 18:00 ladsgroup@deploy2002: ladsgroup: Continuing with sync
* 17:58 ladsgroup@deploy2002: ladsgroup: Backport for gerrit:1115466file: Remove from filerevision when only one row exists (T384481) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 17:55 ladsgroup@deploy2002: Started scap sync-world: Backport for gerrit:1115466file: Remove from filerevision when only one row exists (T384481)
* 17:20 jayme: staging-codfw k8s cluster is currently being updated to k8s 1.31 and in an unusable state - T384450
* 17:19 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2227 (T384592)', diff saved to https://phabricator.wikimedia.org/P72890 and previous config saved to /var/cache/conftool/dbconfig/20250130-171903-marostegui.json
* 17:18 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2227.codfw.wmnet with reason: Maintenance
* 17:18 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2205 (T384592)', diff saved to https://phabricator.wikimedia.org/P72889 and previous config saved to /var/cache/conftool/dbconfig/20250130-171841-marostegui.json
* 17:06 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 17:06 cdanis@deploy2002: helmfile [codfw] START helmfile.d/admin 'apply'.
* 17:04 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 17:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2205', diff saved to https://phabricator.wikimedia.org/P72888 and previous config saved to /var/cache/conftool/dbconfig/20250130-170334-marostegui.json
* 17:02 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 16:48 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2205', diff saved to https://phabricator.wikimedia.org/P72887 and previous config saved to /var/cache/conftool/dbconfig/20250130-164828-marostegui.json
* 16:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2205 (T384592)', diff saved to https://phabricator.wikimedia.org/P72885 and previous config saved to /var/cache/conftool/dbconfig/20250130-163321-marostegui.json
* 16:22 Emperor: repool ms-fe1014 T384317
* 16:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host maps-test2001.codfw.wmnet with OS bookworm
* 16:18 cdanis@deploy2002: Finished scap sync-world: Backport for gerrit:1115436Trace only on k8s. (T321211 T340552 T385037) (duration: 11m 55s)
* 16:10 cdanis@deploy2002: cdanis: Continuing with sync
* 16:09 cdanis@deploy2002: cdanis: Backport for gerrit:1115436Trace only on k8s. (T321211 T340552 T385037) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 16:06 cdanis@deploy2002: Started scap sync-world: Backport for gerrit:1115436Trace only on k8s. (T321211 T340552 T385037)
* 15:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on maps-test2001.codfw.wmnet with reason: host reimage
* 15:54 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on maps-test2001.codfw.wmnet with reason: host reimage
* 15:53 mforns@deploy2002: Finished deploy [airflow-dags/analytics@c85b504]: pin confluent kafka to avoid certificate errors (duration: 00m 52s)
* 15:53 mforns@deploy2002: Started deploy [airflow-dags/analytics@c85b504]: pin confluent kafka to avoid certificate errors
* 
* 15:47 moritzm: installing git security updates
* 15:46 reedy@deploy2002: reedy: Continuing with sync
* 
* 
* 15:41 jmm@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ganeti2029.codfw.wmnet
* 15:41 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ganeti2029.codfw.wmnet
* 15:33 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host maps-test2001.codfw.wmnet with OS bookworm
* 15:30 hashar@deploy2002: Finished scap sync-world: Backport for gerrit:1115103Define new 'auth' docroot with custom files for the auth domain (T383952 T384137) (duration: 14m 55s)
* 15:25 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2029.codfw.wmnet
* 15:23 hashar@deploy2002: matmarex, hashar: Continuing with sync
* 15:18 hashar@deploy2002: matmarex, hashar: Backport for gerrit:1115103Define new 'auth' docroot with custom files for the auth domain (T383952 T384137) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 15:15 hashar@deploy2002: Started scap sync-world: Backport for gerrit:1115103Define new 'auth' docroot with custom files for the auth domain (T383952 T384137)
* 15:11 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host maps-test2001.codfw.wmnet with OS bookworm
* 15:09 ladsgroup@dns1004: END - running authdns-update
* 15:09 hashar@deploy2002: Finished scap sync-world: Backport for gerrit:1113476Use full URLs for wgUploadNavigationUrl (T383916) (duration: 11m 02s)
* 15:07 ladsgroup@dns1004: START - running authdns-update
* 15:04 jmm@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ganeti2029.codfw.wmnet
* 15:03 hashar@deploy2002: hashar, matmarex: Continuing with sync
* 15:03 hashar@deploy2002: hashar, matmarex: Backport for gerrit:1113476Use full URLs for wgUploadNavigationUrl (T383916) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 15:02 jayme: enabled puppet on all kubernetes hosts
* 15:01 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2029.codfw.wmnet with reason: remove from cluster for reimage
* 14:58 hashar@deploy2002: Started scap sync-world: Backport for gerrit:1113476Use full URLs for wgUploadNavigationUrl (T383916)
* 14:56 marostegui@cumin1002: dbctl commit (dc=all): 'db2172 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72882 and previous config saved to /var/cache/conftool/dbconfig/20250130-145620-root.json
* 14:55 elukey@deploy2002: helmfile [eqiad] DONE helmfile.d/services/kartotherian: sync
* 14:54 elukey@deploy2002: helmfile [eqiad] START helmfile.d/services/kartotherian: sync
* 14:51 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2205 (T384592)', diff saved to https://phabricator.wikimedia.org/P72881 and previous config saved to /var/cache/conftool/dbconfig/20250130-145136-marostegui.json
* 14:51 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2205.codfw.wmnet with reason: Maintenance
* 14:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194 (T384592)', diff saved to https://phabricator.wikimedia.org/P72880 and previous config saved to /var/cache/conftool/dbconfig/20250130-145114-marostegui.json
* 14:51 hashar@deploy2002: Finished scap sync-world: Backport for gerrit:1115383SuggestedEditSession: remove incorrect cast to integer (T385117), gerrit:1115384SuggestedEditSession: remove incorrect cast to integer (T385117) (duration: 13m 41s)
* 14:44 hashar@deploy2002: hashar, sgimeno: Continuing with sync
* 14:44 hashar@deploy2002: hashar, sgimeno: Backport for gerrit:1115383SuggestedEditSession: remove incorrect cast to integer (T385117), gerrit:1115384SuggestedEditSession: remove incorrect cast to integer (T385117) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:43 aokoth@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:43 aokoth@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:41 marostegui@cumin1002: dbctl commit (dc=all): 'db2172 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72877 and previous config saved to /var/cache/conftool/dbconfig/20250130-144115-root.json
* 14:37 hashar@deploy2002: Started scap sync-world: Backport for gerrit:1115383SuggestedEditSession: remove incorrect cast to integer (T385117), gerrit:1115384SuggestedEditSession: remove incorrect cast to integer (T385117)
* 14:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P72876 and previous config saved to /var/cache/conftool/dbconfig/20250130-143607-marostegui.json
* 14:32 aokoth@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:26 marostegui@cumin1002: dbctl commit (dc=all): 'db2172 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72875 and previous config saved to /var/cache/conftool/dbconfig/20250130-142609-root.json
* 14:22 aokoth@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P72873 and previous config saved to /var/cache/conftool/dbconfig/20250130-142100-marostegui.json
* 14:20 aokoth@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:19 jayme: stopped puppet on all kubernetes hosts
* 14:18 hashar@deploy2002: Finished scap sync-world: Backport for gerrit:1115062migrateConfigToCommunity: Handle false BabelMainCategory (T384941), gerrit:1115059migrateConfigToCommunity: Handle false BabelMainCategory (T384941), gerrit:1115336migrateConfigToCommunity: Include an edit summary (T385024), gerrit:1115337migrateConfigToCommunity: Include an edit summary (T385024) (duration: 11m 19s)
* 14:11 hashar@deploy2002: urbanecm, hashar: Continuing with sync
* 14:11 marostegui@cumin1002: dbctl commit (dc=all): 'db2172 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72872 and previous config saved to /var/cache/conftool/dbconfig/20250130-141104-root.json
* 14:10 aokoth@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:10 hashar@deploy2002: urbanecm, hashar: Backport for gerrit:1115062migrateConfigToCommunity: Handle false BabelMainCategory (T384941), gerrit:1115059migrateConfigToCommunity: Handle false BabelMainCategory (T384941), gerrit:1115336migrateConfigToCommunity: Include an edit summary (T385024), gerrit:1115337migrateConfigToCommunity: Include an edit summary (T385024) synced to the testservers (https://wik
* 14:06 hashar@deploy2002: Started scap sync-world: Backport for gerrit:1115062migrateConfigToCommunity: Handle false BabelMainCategory (T384941), gerrit:1115059migrateConfigToCommunity: Handle false BabelMainCategory (T384941), gerrit:1115336migrateConfigToCommunity: Include an edit summary (T385024), gerrit:1115337migrateConfigToCommunity: Include an edit summary (T385024)
* 14:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194 (T384592)', diff saved to https://phabricator.wikimedia.org/P72871 and previous config saved to /var/cache/conftool/dbconfig/20250130-140553-marostegui.json
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2029.codfw.wmnet
* 13:55 marostegui@cumin1002: dbctl commit (dc=all): 'db2172 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72870 and previous config saved to /var/cache/conftool/dbconfig/20250130-135559-root.json
* 13:50 jayme@cumin1002: START - Cookbook sre.k8s.wipe-cluster Wipe the K8s cluster staging-codfw: Kubernetes upgrade
* 13:50 marostegui@cumin1002: dbctl commit (dc=all): 'db2224 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72869 and previous config saved to /var/cache/conftool/dbconfig/20250130-135024-root.json
* 13:45 elukey@deploy2002: helmfile [eqiad] DONE helmfile.d/services/kartotherian: sync
* 13:44 elukey@deploy2002: helmfile [eqiad] START helmfile.d/services/kartotherian: sync
* 13:36 hashar@deploy2002: Finished scap sync-world: Backport for gerrit:1115344Fix response error handling in FlickrBlacklist (T385143) (duration: 11m 54s)
* 13:35 marostegui@cumin1002: dbctl commit (dc=all): 'db2224 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72868 and previous config saved to /var/cache/conftool/dbconfig/20250130-133519-root.json
* 13:30 hashar@deploy2002: hashar: Continuing with sync
* 13:28 hashar@deploy2002: hashar: Backport for gerrit:1115344Fix response error handling in FlickrBlacklist (T385143) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 13:24 hashar@deploy2002: Started scap sync-world: Backport for gerrit:1115344Fix response error handling in FlickrBlacklist (T385143)
* 13:20 marostegui@cumin1002: dbctl commit (dc=all): 'db2224 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72867 and previous config saved to /var/cache/conftool/dbconfig/20250130-132014-root.json
* 13:15 aokoth@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 13:09 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on maps-test2001.codfw.wmnet with reason: host reimage
* 13:06 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on maps-test2001.codfw.wmnet with reason: host reimage
* 13:05 marostegui@cumin1002: dbctl commit (dc=all): 'db2224 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72866 and previous config saved to /var/cache/conftool/dbconfig/20250130-130509-root.json
* 13:05 aokoth@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 13:03 jayme@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on 7 hosts with reason: K8s update
* 12:50 marostegui@cumin1002: dbctl commit (dc=all): 'db2224 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72865 and previous config saved to /var/cache/conftool/dbconfig/20250130-125004-root.json
* 12:45 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host maps-test2001.codfw.wmnet with OS bookworm
* 12:28 marostegui@cumin1002: dbctl commit (dc=all): 'db2191 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72864 and previous config saved to /var/cache/conftool/dbconfig/20250130-122856-root.json
* 12:24 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2194 (T384592)', diff saved to https://phabricator.wikimedia.org/P72863 and previous config saved to /var/cache/conftool/dbconfig/20250130-122416-marostegui.json
* 12:24 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2194.codfw.wmnet with reason: Maintenance
* 12:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190 (T384592)', diff saved to https://phabricator.wikimedia.org/P72862 and previous config saved to /var/cache/conftool/dbconfig/20250130-122354-marostegui.json
* 12:13 marostegui@cumin1002: dbctl commit (dc=all): 'db2191 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72861 and previous config saved to /var/cache/conftool/dbconfig/20250130-121351-root.json
* 12:11 hnowlan@deploy2002: Finished scap sync-world: testing removal of scap proxies (duration: 03m 18s)
* 12:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P72860 and previous config saved to /var/cache/conftool/dbconfig/20250130-120847-marostegui.json
* 12:08 hnowlan@deploy2002: Started scap sync-world: testing removal of scap proxies
* 11:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2191 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72859 and previous config saved to /var/cache/conftool/dbconfig/20250130-115846-root.json
* 11:53 marostegui@cumin1002: dbctl commit (dc=all): 'db1183 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72858 and previous config saved to /var/cache/conftool/dbconfig/20250130-115348-root.json
* 11:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P72857 and previous config saved to /var/cache/conftool/dbconfig/20250130-115340-marostegui.json
* 11:52 elukey@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 11:43 marostegui@cumin1002: dbctl commit (dc=all): 'db2191 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72856 and previous config saved to /var/cache/conftool/dbconfig/20250130-114341-root.json
* 11:40 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2029.codfw.wmnet
* 11:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2029.codfw.wmnet
* 11:39 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2029.codfw.wmnet
* 11:38 marostegui@cumin1002: dbctl commit (dc=all): 'db1183 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72855 and previous config saved to /var/cache/conftool/dbconfig/20250130-113842-root.json
* 11:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190 (T384592)', diff saved to https://phabricator.wikimedia.org/P72854 and previous config saved to /var/cache/conftool/dbconfig/20250130-113833-marostegui.json
* 11:35 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2224.codfw.wmnet with reason: Index rebuild
* 11:34 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2224.codfw.wmnet
* 11:29 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2224.codfw.wmnet
* 11:28 ladsgroup@deploy2002: Synchronized portals/wikipedia.org/assets: Bump portals (second try) (duration: 11m 07s)
* 11:28 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2224', diff saved to https://phabricator.wikimedia.org/P72853 and previous config saved to /var/cache/conftool/dbconfig/20250130-112853-marostegui.json
* 11:28 marostegui@cumin1002: dbctl commit (dc=all): 'db2191 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72852 and previous config saved to /var/cache/conftool/dbconfig/20250130-112836-root.json
* 11:28 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2191.codfw.wmnet
* 11:23 marostegui@cumin1002: dbctl commit (dc=all): 'db1183 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72851 and previous config saved to /var/cache/conftool/dbconfig/20250130-112337-root.json
* 11:22 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2191.codfw.wmnet
* 11:22 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2191', diff saved to https://phabricator.wikimedia.org/P72850 and previous config saved to /var/cache/conftool/dbconfig/20250130-112252-marostegui.json
* 11:22 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2172.codfw.wmnet with reason: Index rebuild
* 11:21 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2172.codfw.wmnet
* 11:17 ladsgroup@deploy2002: Synchronized portals/wikipedia.org/assets: Bump portals (T368221 and T373204) (duration: 02m 57s)
* 11:16 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1153.eqiad.wmnet
* 11:13 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2172.codfw.wmnet
* 11:12 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2172', diff saved to https://phabricator.wikimedia.org/P72849 and previous config saved to /var/cache/conftool/dbconfig/20250130-111244-marostegui.json
* 11:10 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1153.eqiad.wmnet
* 11:09 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2144.codfw.wmnet
* 11:09 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1151.eqiad.wmnet
* 11:08 marostegui@cumin1002: dbctl commit (dc=all): 'db1183 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72848 and previous config saved to /var/cache/conftool/dbconfig/20250130-110832-root.json
* 11:03 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2144.codfw.wmnet
* 11:02 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1151.eqiad.wmnet
* 10:57 elukey@deploy2002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
* 10:56 elukey@deploy2002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
* 10:53 marostegui@cumin1002: dbctl commit (dc=all): 'db1183 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72847 and previous config saved to /var/cache/conftool/dbconfig/20250130-105327-root.json
* 10:44 moritzm: installing util-linux bugfix updates from bookworm point release
* 10:39 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2201.codfw.wmnet
* 10:39 jynus@cumin1002: START - Cookbook sre.hosts.remove-downtime for db2201.codfw.wmnet
* 10:38 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2202.codfw.wmnet with reason: prepare for decom
* 10:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2201.codfw.wmnet with reason: test new s4 backups
* 10:19 cmooney@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host frbast1002
* 10:19 cmooney@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host frbast1002
* 10:16 cmooney@cumin1002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host frbast1002
* 10:16 cmooney@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host frbast1002
* 10:04 cmooney@cumin1002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host frbast1002
* 10:04 cmooney@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host frbast1002
* 09:57 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2030.codfw.wmnet to cluster codfw and group A
* 09:57 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2190 (T384592)', diff saved to https://phabricator.wikimedia.org/P72846 and previous config saved to /var/cache/conftool/dbconfig/20250130-095706-marostegui.json
* 09:57 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2190.codfw.wmnet with reason: Maintenance
* 09:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T384592)', diff saved to https://phabricator.wikimedia.org/P72845 and previous config saved to /var/cache/conftool/dbconfig/20250130-095644-marostegui.json
* 09:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2030.codfw.wmnet to cluster codfw and group A
* 09:56 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2039.codfw.wmnet to cluster codfw and group A
* 09:56 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2039.codfw.wmnet to cluster codfw and group A
* 09:55 marostegui@cumin1002: dbctl commit (dc=all): 'db1161 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72844 and previous config saved to /var/cache/conftool/dbconfig/20250130-095524-root.json
* 09:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2030.codfw.wmnet
* 09:46 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1183.eqiad.wmnet with reason: Index rebuild
* 09:46 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2030.codfw.wmnet
* 09:46 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1183.eqiad.wmnet
* 09:43 moritzm: removed profile::ssh::server::disable_nist_kex: false from Toolforge Hiera settings (so that the defaults apply like for the rest of Cloud VPS)
* 09:41 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2030.codfw.wmnet with OS bookworm
* 09:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P72843 and previous config saved to /var/cache/conftool/dbconfig/20250130-094137-marostegui.json
* 09:40 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1183.eqiad.wmnet
* 09:40 marostegui@cumin1002: dbctl commit (dc=all): 'db1161 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72842 and previous config saved to /var/cache/conftool/dbconfig/20250130-094018-root.json
* 09:39 elukey@deploy2002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
* 09:39 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1183 T385147', diff saved to https://phabricator.wikimedia.org/P72841 and previous config saved to /var/cache/conftool/dbconfig/20250130-093927-marostegui.json
* 09:38 marostegui@cumin2002: dbctl commit (dc=all): 'Promote db1230 to s5 primary T385147', diff saved to https://phabricator.wikimedia.org/P72840 and previous config saved to /var/cache/conftool/dbconfig/20250130-093845-marostegui.json
* 09:38 elukey@deploy2002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
* 09:38 marostegui: Starting s5 eqiad failover from db1183 to db1230 - T385147
* 09:32 marostegui@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s5 T385147
* 09:32 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1230 with weight 0 T385147', diff saved to https://phabricator.wikimedia.org/P72839 and previous config saved to /var/cache/conftool/dbconfig/20250130-093221-root.json
* 09:28 elukey@deploy2002: helmfile [staging] DONE helmfile.d/services/kartotherian: sync
* 09:28 elukey@deploy2002: helmfile [staging] START helmfile.d/services/kartotherian: sync
* 09:26 elukey@deploy2002: helmfile [staging] DONE helmfile.d/services/kartotherian: sync
* 09:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P72838 and previous config saved to /var/cache/conftool/dbconfig/20250130-092630-marostegui.json
* 09:26 elukey@deploy2002: helmfile [staging] START helmfile.d/services/kartotherian: sync
* 09:25 marostegui@cumin1002: dbctl commit (dc=all): 'db1161 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72837 and previous config saved to /var/cache/conftool/dbconfig/20250130-092513-root.json
* 09:21 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2030.codfw.wmnet with reason: host reimage
* 09:18 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2030.codfw.wmnet with reason: host reimage
* 09:16 hashar@deploy2002: rebuilt and synchronized wikiversions files: group1 to 1.44.0-wmf.14 refs T382365
* 09:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T384592)', diff saved to https://phabricator.wikimedia.org/P72836 and previous config saved to /var/cache/conftool/dbconfig/20250130-091123-marostegui.json
* 09:10 marostegui@cumin1002: dbctl commit (dc=all): 'db1161 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72835 and previous config saved to /var/cache/conftool/dbconfig/20250130-091007-root.json
* 09:08 elukey@deploy2002: helmfile [staging] DONE helmfile.d/services/kartotherian: sync
* 09:08 elukey@deploy2002: helmfile [staging] START helmfile.d/services/kartotherian: sync
* 08:55 marostegui@cumin1002: dbctl commit (dc=all): 'db1161 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72834 and previous config saved to /var/cache/conftool/dbconfig/20250130-085502-root.json
* 08:49 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2030.codfw.wmnet with OS bookworm
* 08:48 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti2030.codfw.wmnet with OS bookworm
* 08:45 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2095,2175,2186].codfw.wmnet
* 08:45 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker[2095,2175,2186].codfw.wmnet
* 08:45 jayme@cumin1002: START - Cookbook sre.hosts.remove-downtime for wikikube-worker[2095,2175,2186].codfw.wmnet
* 08:45 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2095,2175,2186].codfw.wmnet
* 08:33 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2030.codfw.wmnet with OS bookworm
* 08:25 jmm@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ganeti2030.codfw.wmnet
* 08:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2030.codfw.wmnet
* 08:14 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2030.codfw.wmnet
* 08:09 marostegui@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Rebuild and upgrade dbstore1007:s4
* 07:49 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1161.eqiad.wmnet with reason: Index rebuild
* 07:48 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1161.eqiad.wmnet
* 07:47 marostegui@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on an-redacteddb1001.eqiad.wmnet with reason: Rebuild and upgrade db1166
* 07:46 marostegui@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1016,1020].eqiad.wmnet with reason: Rebuild and upgrade db1166
* 07:45 marostegui@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1154.eqiad.wmnet with reason: Rebuild and upgrade db1166
* 07:42 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1161.eqiad.wmnet
* 07:42 jmm@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ganeti2030.codfw.wmnet
* 07:42 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1161 T384994', diff saved to https://phabricator.wikimedia.org/P72833 and previous config saved to /var/cache/conftool/dbconfig/20250130-074228-marostegui.json
* 07:38 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2031.codfw.wmnet to cluster codfw and group B
* 07:36 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2031.codfw.wmnet to cluster codfw and group B
* 07:30 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2030.codfw.wmnet with reason: remove from cluster for reimage
* 07:24 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2177 (T384592)', diff saved to https://phabricator.wikimedia.org/P72832 and previous config saved to /var/cache/conftool/dbconfig/20250130-072432-marostegui.json
* 07:24 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2177.codfw.wmnet with reason: Maintenance
* 07:24 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T384592)', diff saved to https://phabricator.wikimedia.org/P72831 and previous config saved to /var/cache/conftool/dbconfig/20250130-072410-marostegui.json
* 07:13 vgutierrez: repooling lvs4009 after reimaging as a liberica load balancer - T384477
* 07:09 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P72830 and previous config saved to /var/cache/conftool/dbconfig/20250130-070903-marostegui.json
* 06:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P72829 and previous config saved to /var/cache/conftool/dbconfig/20250130-065356-marostegui.json
* 06:41 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs4009.ulsfo.wmnet with OS bookworm
* 06:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T384592)', diff saved to https://phabricator.wikimedia.org/P72828 and previous config saved to /var/cache/conftool/dbconfig/20250130-063849-marostegui.json
* 06:35 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1251.eqiad.wmnet with OS bookworm
* 06:35 vriley@cumin1002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1002"
* 06:35 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1250.eqiad.wmnet with OS bookworm
* 06:35 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1002"
* 06:17 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4009.ulsfo.wmnet with reason: host reimage
* 06:14 vgutierrez@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4009.ulsfo.wmnet with reason: host reimage
* 05:57 vgutierrez@cumin1002: START - Cookbook sre.hosts.reimage for host lvs4009.ulsfo.wmnet with OS bookworm
* 05:19 vgutierrez: depooling lvs4009 prior to reimaging as a liberica load balancer - T384477
* 04:51 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2156 (T384592)', diff saved to https://phabricator.wikimedia.org/P72827 and previous config saved to /var/cache/conftool/dbconfig/20250130-045147-marostegui.json
* 04:51 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2186.codfw.wmnet with reason: Maintenance
* 04:51 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance
* 04:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T384592)', diff saved to https://phabricator.wikimedia.org/P72826 and previous config saved to /var/cache/conftool/dbconfig/20250130-045108-marostegui.json
* 04:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P72825 and previous config saved to /var/cache/conftool/dbconfig/20250130-043601-marostegui.json
* 04:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P72824 and previous config saved to /var/cache/conftool/dbconfig/20250130-042054-marostegui.json
* 04:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T384592)', diff saved to https://phabricator.wikimedia.org/P72823 and previous config saved to /var/cache/conftool/dbconfig/20250130-040548-marostegui.json
* 02:45 swfrench-wmf: scaled down shellbox-video/migration after switch to PHP 8.1 - T377038
* 02:44 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
* 02:44 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-video: apply
* 02:16 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2149 (T384592)', diff saved to https://phabricator.wikimedia.org/P72822 and previous config saved to /var/cache/conftool/dbconfig/20250130-021645-marostegui.json
* 02:16 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2149.codfw.wmnet with reason: Maintenance
== 2025-01-29 ==
* 23:53 ejegg: fundraising civicrm upgraded from to 
* 23:50 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2139.codfw.wmnet with reason: Maintenance
* 23:45 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
* 23:44 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
* 23:43 cmooney@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host backup1010
* 23:43 cmooney@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host backup1010
* 23:41 cmooney@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db1251
* 23:39 cmooney@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host db1251
* 23:30 cmooney@cumin1002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host db1251
* 23:28 cmooney@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host db1251
* 23:21 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
* 23:20 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-video: apply
* 23:18 cmooney@cumin1002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host db1251
* 23:17 pt1979@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 23:12 pt1979@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 23:11 cmooney@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host db1251
* 23:09 cmooney@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2186
* 23:09 cmooney@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2186
* 23:06 cmooney@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2095
* 23:05 cmooney@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2095
* 22:54 rzl@deploy2002: Finished scap sync-world: T357595 (duration: 11m 57s)
* 22:49 rzl@deploy2002: rzl: Continuing with sync
* 22:46 rzl@deploy2002: rzl: T357595 synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 22:43 rzl@deploy2002: Started scap sync-world: T357595
* 22:27 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1002"
* 22:06 vriley@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1251.eqiad.wmnet with reason: host reimage
* 22:02 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1250.eqiad.wmnet with reason: host reimage
* 21:55 vriley@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1250.eqiad.wmnet with reason: host reimage
* 21:52 catrope@deploy2002: Finished scap sync-world: Backport for gerrit:1115099resourceloader: Fix hash computation for virtual files with versionFilePath (T385055), gerrit:1115098resourceloader: Fix hash computation for virtual files with versionFilePath (T385055) (duration: 17m 29s)
* 21:50 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host db1251.eqiad.wmnet with OS bookworm
* 21:48 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1251.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 21:45 catrope@deploy2002: cdanis, catrope: Continuing with sync
* 21:42 catrope@deploy2002: cdanis, catrope: Backport for gerrit:1115099resourceloader: Fix hash computation for virtual files with versionFilePath (T385055), gerrit:1115098resourceloader: Fix hash computation for virtual files with versionFilePath (T385055) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:40 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host db1250.eqiad.wmnet with OS bookworm
* 21:35 catrope@deploy2002: Started scap sync-world: Backport for gerrit:1115099resourceloader: Fix hash computation for virtual files with versionFilePath (T385055), gerrit:1115098resourceloader: Fix hash computation for virtual files with versionFilePath (T385055)
* 21:15 vriley@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['db1250']
* 21:13 vriley@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db1250']
* 21:12 vriley@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['db1250']
* 21:11 vriley@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db1250']
* 21:11 vriley@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['db1250']
* 21:10 vriley@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db1250']
* 21:10 vriley@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['db1250']
* 21:09 vriley@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db1250']
* 21:07 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1250.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 20:58 pt1979@cumin1002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['ms-fe1014']
* 20:51 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt2006-dev.codfw.wmnet
* 20:44 andrew@cumin1002: START - Cookbook sre.hosts.reboot-single for host cloudvirt2006-dev.codfw.wmnet
* 20:40 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1240.eqiad.wmnet with reason: Maintenance
* 20:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223 (T384592)', diff saved to https://phabricator.wikimedia.org/P72818 and previous config saved to /var/cache/conftool/dbconfig/20250129-204020-marostegui.json
* 20:32 pt1979@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-fe1014']
* 20:32 vriley@cumin1002: START - Cookbook sre.hosts.provision for host db1251.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 20:32 pt1979@cumin1002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['ms-fe1014']
* 20:30 vriley@cumin1002: START - Cookbook sre.hosts.provision for host db1250.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 20:27 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 20:25 vriley@cumin1002: START - Cookbook sre.dns.netbox
* 20:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P72817 and previous config saved to /var/cache/conftool/dbconfig/20250129-202513-marostegui.json
* 20:25 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 20:25 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt db1250 - vriley@cumin1002"
* 20:24 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt db1250 - vriley@cumin1002"
* 20:21 vriley@cumin1002: START - Cookbook sre.dns.netbox
* 20:16 vriley@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 20:14 vriley@cumin1002: START - Cookbook sre.dns.netbox
* 20:13 pt1979@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-fe1014']
* 20:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P72816 and previous config saved to /var/cache/conftool/dbconfig/20250129-201006-marostegui.json
* 20:03 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 20:00 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 19:58 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 19:56 pt1979@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ms-fe1014']
* 19:56 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 19:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223 (T384592)', diff saved to https://phabricator.wikimedia.org/P72815 and previous config saved to /var/cache/conftool/dbconfig/20250129-195459-marostegui.json
* 19:54 inflatador: bking@apt1002 publish new opensearch_1.3.20 pkg to thirdparty/opensearch1
* 19:49 pt1979@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-fe1014']
* 19:48 xcollazo@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
* 19:48 xcollazo@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
* 19:47 pt1979@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ms-fe1014']
* 19:44 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 19:43 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 19:42 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 19:34 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ganeti1053.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 19:26 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1223 (T384592)', diff saved to https://phabricator.wikimedia.org/P72814 and previous config saved to /var/cache/conftool/dbconfig/20250129-192637-marostegui.json
* 19:26 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1223.eqiad.wmnet with reason: Maintenance
* 19:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T384592)', diff saved to https://phabricator.wikimedia.org/P72813 and previous config saved to /var/cache/conftool/dbconfig/20250129-192615-marostegui.json
* 19:20 pt1979@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-fe1014']
* 19:19 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 19:17 vriley@cumin1002: START - Cookbook sre.dns.netbox
* 19:16 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 19:16 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt ganeti1053 - vriley@cumin1002"
* 19:16 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt ganeti1053 - vriley@cumin1002"
* 19:15 otto@deploy2002: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics-external: apply
* 19:14 otto@deploy2002: helmfile [eqiad] START helmfile.d/services/eventgate-analytics-external: apply
* 19:14 otto@deploy2002: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: apply
* 19:13 otto@deploy2002: helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: apply
* 19:12 vriley@cumin1002: START - Cookbook sre.dns.netbox
* 19:12 marostegui@cumin1002: dbctl commit (dc=all): 'db1159 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72812 and previous config saved to /var/cache/conftool/dbconfig/20250129-191234-root.json
* 19:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P72811 and previous config saved to /var/cache/conftool/dbconfig/20250129-191108-marostegui.json
* 19:09 otto@deploy2002: helmfile [staging] DONE helmfile.d/services/eventgate-analytics-external: apply
* 19:09 otto@deploy2002: helmfile [staging] START helmfile.d/services/eventgate-analytics-external: apply
* 18:57 marostegui@cumin1002: dbctl commit (dc=all): 'db1159 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72810 and previous config saved to /var/cache/conftool/dbconfig/20250129-185729-root.json
* 18:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P72809 and previous config saved to /var/cache/conftool/dbconfig/20250129-185602-marostegui.json
* 18:53 aokoth@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 18:45 xcollazo@deploy2002: Finished deploy [airflow-dags/analytics@5b0aeae]: Deploying latest DAGs to the analytics Airflow instance. T358375. (duration: 00m 35s)
* 18:44 xcollazo@deploy2002: Started deploy [airflow-dags/analytics@5b0aeae]: Deploying latest DAGs to the analytics Airflow instance. T358375.
* 18:43 aokoth@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 18:42 marostegui@cumin1002: dbctl commit (dc=all): 'db1159 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72805 and previous config saved to /var/cache/conftool/dbconfig/20250129-184223-root.json
* 18:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T384592)', diff saved to https://phabricator.wikimedia.org/P72804 and previous config saved to /var/cache/conftool/dbconfig/20250129-184055-marostegui.json
* 18:37 aokoth@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 18:27 marostegui@cumin1002: dbctl commit (dc=all): 'db1159 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72801 and previous config saved to /var/cache/conftool/dbconfig/20250129-182718-root.json
* 18:26 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2030.codfw.wmnet
* 18:26 aokoth@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 18:13 cdanis: and https://... too because idk if it matters tbh
* 18:13 cdanis: 💙cdanis@mwmaint2002.codfw.wmnet ~ 🕐☕ echo 'http://www.mediawiki.org/w/load.php?lang=en&modules=vue&skin=vector-2022&version=n99vv' mwscript purgeList.php
* 18:12 marostegui@cumin1002: dbctl commit (dc=all): 'db1159 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72796 and previous config saved to /var/cache/conftool/dbconfig/20250129-181212-root.json
* 18:10 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1212 (T384592)', diff saved to https://phabricator.wikimedia.org/P72795 and previous config saved to /var/cache/conftool/dbconfig/20250129-181037-marostegui.json
* 18:10 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 18:10 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1212.eqiad.wmnet with reason: Maintenance
* 18:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T384592)', diff saved to https://phabricator.wikimedia.org/P72794 and previous config saved to /var/cache/conftool/dbconfig/20250129-181017-marostegui.json
* 17:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P72792 and previous config saved to /var/cache/conftool/dbconfig/20250129-175510-marostegui.json
* 17:54 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: maintenance
* 17:53 aokoth@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 17:50 hnowlan@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2242-2243].codfw.wmnet
* 17:50 hnowlan@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2242-2243].codfw.wmnet
* 17:48 hnowlan: homer 'lsw1-a5-codfw*' commit
* 17:45 hnowlan@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2243.codfw.wmnet with OS bookworm
* 17:42 aokoth@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 17:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P72790 and previous config saved to /var/cache/conftool/dbconfig/20250129-174003-marostegui.json
* 17:38 jgiannelos@deploy2002: helmfile [staging] DONE helmfile.d/services/kartotherian: apply
* 17:38 jgiannelos@deploy2002: helmfile [staging] START helmfile.d/services/kartotherian: apply
* 17:37 jgiannelos@deploy2002: helmfile [staging] START helmfile.d/services/kartotherian: apply
* 17:37 jgiannelos@deploy2002: helmfile [staging] DONE helmfile.d/services/kartotherian: apply
* 17:37 jgiannelos@deploy2002: helmfile [staging] START helmfile.d/services/kartotherian: apply
* 17:37 hnowlan@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2242.codfw.wmnet with OS bookworm
* 17:37 jgiannelos@deploy2002: helmfile [staging] DONE helmfile.d/services/kartotherian: apply
* 17:37 jgiannelos@deploy2002: helmfile [staging] START helmfile.d/services/kartotherian: apply
* 17:37 jgiannelos@deploy2002: helmfile [staging] DONE helmfile.d/services/kartotherian: apply
* 17:36 jgiannelos@deploy2002: helmfile [staging] START helmfile.d/services/kartotherian: apply
* 17:36 jgiannelos@deploy2002: helmfile [staging] DONE helmfile.d/services/kartotherian: apply
* 17:36 jgiannelos@deploy2002: helmfile [staging] START helmfile.d/services/kartotherian: apply
* 17:27 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2186
* 17:27 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2186
* 17:26 hnowlan@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2243.codfw.wmnet with reason: host reimage
* 17:24 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T384592)', diff saved to https://phabricator.wikimedia.org/P72789 and previous config saved to /var/cache/conftool/dbconfig/20250129-172455-marostegui.json
* 17:23 hnowlan@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2243.codfw.wmnet with reason: host reimage
* 17:20 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2175
* 17:20 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2175
* 17:17 hnowlan@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2242.codfw.wmnet with reason: host reimage
* 17:14 hnowlan@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2242.codfw.wmnet with reason: host reimage
* 17:07 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1159.eqiad.wmnet with reason: Index rebuild
* 17:06 hnowlan@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2243
* 17:06 hnowlan@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2243
* 17:05 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1159.eqiad.wmnet
* 17:04 hnowlan@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2243
* 17:04 hnowlan@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2243.codfw.wmnet 122.0.192.10.in-addr.arpa 2.2.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 17:03 hnowlan@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2243.codfw.wmnet 122.0.192.10.in-addr.arpa 2.2.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 17:03 hnowlan@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:03 hnowlan@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2243 - hnowlan@cumin2002"
* 17:03 hnowlan@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2243 - hnowlan@cumin2002"
* 17:00 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1159.eqiad.wmnet
* 16:59 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1159 T384994', diff saved to https://phabricator.wikimedia.org/P72787 and previous config saved to /var/cache/conftool/dbconfig/20250129-165951-marostegui.json
* 16:59 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1198 (T384592)', diff saved to https://phabricator.wikimedia.org/P72786 and previous config saved to /var/cache/conftool/dbconfig/20250129-165935-marostegui.json
* 16:59 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1198.eqiad.wmnet with reason: Maintenance
* 16:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T384592)', diff saved to https://phabricator.wikimedia.org/P72785 and previous config saved to /var/cache/conftool/dbconfig/20250129-165913-marostegui.json
* 16:58 hnowlan@cumin2002: START - Cookbook sre.dns.netbox
* 16:58 hnowlan@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2243
* 16:58 hnowlan@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2243.codfw.wmnet with OS bookworm
* 16:58 hnowlan@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2242
* 16:58 hnowlan@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2242
* 16:57 hnowlan@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2242
* 16:57 hnowlan@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2242.codfw.wmnet 113.0.192.10.in-addr.arpa 3.1.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 16:57 hnowlan@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2243.codfw.wmnet on all recursors
* 16:57 hnowlan@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2242.codfw.wmnet 113.0.192.10.in-addr.arpa 3.1.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 16:57 hnowlan@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:57 hnowlan@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2242 - hnowlan@cumin2002"
* 16:57 hnowlan@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2243.codfw.wmnet on all recursors
* 16:57 hnowlan@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2242 - hnowlan@cumin2002"
* 16:56 hnowlan@cumin2002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2411 to wikikube-worker2243
* 16:56 elukey@deploy2002: helmfile [staging] DONE helmfile.d/services/kartotherian: sync
* 16:56 hnowlan@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2243
* 16:56 elukey@deploy2002: helmfile [staging] START helmfile.d/services/kartotherian: sync
* 16:51 jayme@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wikikube-worker[2095,2175,2186].codfw.wmnet with reason: extending downtime
* 16:51 hnowlan@cumin2002: START - Cookbook sre.dns.netbox
* 16:51 hnowlan@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2243
* 16:51 hnowlan@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:51 hnowlan@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2411 to wikikube-worker2243 - hnowlan@cumin2002"
* 16:51 hnowlan@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2411 to wikikube-worker2243 - hnowlan@cumin2002"
* 16:48 hnowlan@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2242
* 16:48 hnowlan@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2242.codfw.wmnet with OS bookworm
* 16:47 hnowlan@cumin2002: START - Cookbook sre.dns.netbox
* 16:47 hnowlan@cumin2002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2410 to wikikube-worker2242
* 16:46 hnowlan@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2242
* 16:46 hnowlan@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2242
* 16:46 hnowlan@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:46 hnowlan@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2410 to wikikube-worker2242 - hnowlan@cumin2002"
* 16:46 hnowlan@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2410 to wikikube-worker2242 - hnowlan@cumin2002"
* 16:44 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P72784 and previous config saved to /var/cache/conftool/dbconfig/20250129-164406-marostegui.json
* 16:41 hnowlan@cumin2002: START - Cookbook sre.dns.netbox
* 16:41 hnowlan@cumin2002: START - Cookbook sre.hosts.rename from mw2411 to wikikube-worker2243
* 16:41 hnowlan@cumin2002: START - Cookbook sre.hosts.rename from mw2410 to wikikube-worker2242
* 16:39 marostegui@cumin1002: dbctl commit (dc=all): 'db2178 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72783 and previous config saved to /var/cache/conftool/dbconfig/20250129-163901-root.json
* 16:37 marostegui@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72782 and previous config saved to /var/cache/conftool/dbconfig/20250129-163729-root.json
* 16:32 moritzm: installing util-linux bugfix updates from bookworm point release
* 16:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P72781 and previous config saved to /var/cache/conftool/dbconfig/20250129-162900-marostegui.json
* 16:28 marostegui@cumin1002: dbctl commit (dc=all): 'db1212 (re)pooling @ 100%: Repooling after rebuild index $TASKID', diff saved to https://phabricator.wikimedia.org/P72780 and previous config saved to /var/cache/conftool/dbconfig/20250129-162822-root.json
* 16:23 marostegui@cumin1002: dbctl commit (dc=all): 'db2178 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72778 and previous config saved to /var/cache/conftool/dbconfig/20250129-162355-root.json
* 16:22 marostegui@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72777 and previous config saved to /var/cache/conftool/dbconfig/20250129-162224-root.json
* 16:22 swfrench@deploy2002: Finished scap sync-world: Backport for gerrit:1114793Enroll 5% of client sessions in PHP 8.1 (T383845) (duration: 15m 00s)
* 16:14 swfrench@deploy2002: swfrench: Continuing with sync
* 16:14 moritzm: installing glib2.0 security updates
* 16:13 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T384592)', diff saved to https://phabricator.wikimedia.org/P72776 and previous config saved to /var/cache/conftool/dbconfig/20250129-161353-marostegui.json
* 16:13 swfrench@deploy2002: swfrench: Backport for gerrit:1114793Enroll 5% of client sessions in PHP 8.1 (T383845) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 16:13 marostegui@cumin1002: dbctl commit (dc=all): 'db1212 (re)pooling @ 75%: Repooling after rebuild index $TASKID', diff saved to https://phabricator.wikimedia.org/P72775 and previous config saved to /var/cache/conftool/dbconfig/20250129-161317-root.json
* 16:08 marostegui@cumin1002: dbctl commit (dc=all): 'db2178 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72774 and previous config saved to /var/cache/conftool/dbconfig/20250129-160850-root.json
* 16:07 marostegui@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72773 and previous config saved to /var/cache/conftool/dbconfig/20250129-160718-root.json
* 16:06 swfrench@deploy2002: Started scap sync-world: Backport for gerrit:1114793Enroll 5% of client sessions in PHP 8.1 (T383845)
* 16:02 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2095
* 16:02 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2095
* 16:00 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 16:00 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2095
* 16:00 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2095
* 16:00 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2095
* 16:00 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:59 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2095
* 15:58 marostegui@cumin1002: dbctl commit (dc=all): 'db1212 (re)pooling @ 50%: Repooling after rebuild index $TASKID', diff saved to https://phabricator.wikimedia.org/P72772 and previous config saved to /var/cache/conftool/dbconfig/20250129-155812-root.json
* 15:54 otto@deploy2002: Finished scap sync-world: Backport for gerrit:1115042EventStreamConfig - prep for per stream user agent collection config (T382173) (duration: 17m 08s)
* 15:53 marostegui@cumin1002: dbctl commit (dc=all): 'db2178 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72771 and previous config saved to /var/cache/conftool/dbconfig/20250129-155345-root.json
* 15:52 marostegui@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72770 and previous config saved to /var/cache/conftool/dbconfig/20250129-155213-root.json
* 15:48 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1175 (T384592)', diff saved to https://phabricator.wikimedia.org/P72769 and previous config saved to /var/cache/conftool/dbconfig/20250129-154807-marostegui.json
* 15:48 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 15:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T384592)', diff saved to https://phabricator.wikimedia.org/P72768 and previous config saved to /var/cache/conftool/dbconfig/20250129-154757-marostegui.json
* 15:47 otto@deploy2002: otto: Continuing with sync
* 15:44 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2031.codfw.wmnet
* 15:43 marostegui@cumin1002: dbctl commit (dc=all): 'db1212 (re)pooling @ 25%: Repooling after rebuild index $TASKID', diff saved to https://phabricator.wikimedia.org/P72767 and previous config saved to /var/cache/conftool/dbconfig/20250129-154306-root.json
* 15:42 otto@deploy2002: otto: Backport for gerrit:1115042EventStreamConfig - prep for per stream user agent collection config (T382173) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 15:38 marostegui@cumin1002: dbctl commit (dc=all): 'db2178 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72766 and previous config saved to /var/cache/conftool/dbconfig/20250129-153840-root.json
* 15:37 otto@deploy2002: Started scap sync-world: Backport for gerrit:1115042EventStreamConfig - prep for per stream user agent collection config (T382173)
* 15:37 marostegui@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72765 and previous config saved to /var/cache/conftool/dbconfig/20250129-153708-root.json
* 15:36 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2031.codfw.wmnet
* 15:33 jmm@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ganeti2031.codfw.wmnet
* 15:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2031.codfw.wmnet
* 15:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P72764 and previous config saved to /var/cache/conftool/dbconfig/20250129-153250-marostegui.json
* 15:28 marostegui@cumin1002: dbctl commit (dc=all): 'db1212 (re)pooling @ 10%: Repooling after rebuild index $TASKID', diff saved to https://phabricator.wikimedia.org/P72763 and previous config saved to /var/cache/conftool/dbconfig/20250129-152801-root.json
* 15:27 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ms-be[1051-1059].eqiad.wmnet
* 15:27 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:27 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ms-be[1051-1059].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - mvernon@cumin2002"
* 15:26 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ms-be[1051-1059].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - mvernon@cumin2002"
* 15:23 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2031.codfw.wmnet
* 15:22 mvernon@cumin2002: START - Cookbook sre.dns.netbox
* 15:19 apine@deploy2002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 15:18 apine@deploy2002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 15:18 apine@deploy2002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 15:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P72762 and previous config saved to /var/cache/conftool/dbconfig/20250129-151743-marostegui.json
* 15:17 apine@deploy2002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 15:16 apine@deploy2002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 15:16 apine@deploy2002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 15:15 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 15:14 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 15:13 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 15:13 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 15:11 apine@deploy2002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 15:10 apine@deploy2002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 15:09 apine@deploy2002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 15:08 apine@deploy2002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 15:06 apine@deploy2002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 15:06 apine@deploy2002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 15:05 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2095,2175,2186].codfw.wmnet
* 15:03 jayme@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on wikikube-worker[2095,2175,2186].codfw.wmnet with reason: Depooled via sre.k8s.pool-depool-node
* 15:03 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2095,2175,2186].codfw.wmnet
* 15:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T384592)', diff saved to https://phabricator.wikimedia.org/P72761 and previous config saved to /var/cache/conftool/dbconfig/20250129-150236-marostegui.json
* 15:02 jynus@cumin1002: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1 day, 0:00:00 on db2201.codfw.wmnet with reason: upgrade kernel and rebuilding tables
* 14:58 Lucas_WMDE: UTC afternoon backport+config window done
* 14:58 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1114995Handle missing `monthonly` format in MwTimeIsoFormatter (T384867), gerrit:1114994Handle missing `monthonly` format in MwTimeIsoFormatter (T384867), gerrit:1115012Handle null date format in MwDateFormatParserFactory (T384963), gerrit:1115010Handle null date format in MwDateFormatParserFactory (T384963) (duration:
* 14:54 vgutierrez: repool ncredir4002
* 14:53 mvernon@cumin2002: START - Cookbook sre.hosts.decommission for hosts ms-be[1051-1059].eqiad.wmnet
* 14:51 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde: Continuing with sync
* 14:50 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde: Backport for gerrit:1114995Handle missing `monthonly` format in MwTimeIsoFormatter (T384867), gerrit:1114994Handle missing `monthonly` format in MwTimeIsoFormatter (T384867), gerrit:1115012Handle null date format in MwDateFormatParserFactory (T384963), gerrit:1115010Handle null date format in MwDateFormatParserFactory (T384963) synced to the
* 14:46 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1114995Handle missing `monthonly` format in MwTimeIsoFormatter (T384867), gerrit:1114994Handle missing `monthonly` format in MwTimeIsoFormatter (T384867), gerrit:1115012Handle null date format in MwDateFormatParserFactory (T384963), gerrit:1115010Handle null date format in MwDateFormatParserFactory (T384963)
* 14:42 aokoth@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 14:42 jmm@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ganeti2031.codfw.wmnet
* 14:41 hnowlan@deploy2002: Finished scap sync-world: Backport for gerrit:1114398fc-list: update font list (T280718) (duration: 16m 00s)
* 14:41 aokoth@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 14:40 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2178.codfw.wmnet with reason: Index rebuild
* 14:40 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1185.eqiad.wmnet with reason: Index rebuild
* 14:40 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2178.codfw.wmnet
* 14:39 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1185.eqiad.wmnet
* 14:35 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1166 (T384592)', diff saved to https://phabricator.wikimedia.org/P72758 and previous config saved to /var/cache/conftool/dbconfig/20250129-143510-marostegui.json
* 14:35 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 14:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T384592)', diff saved to https://phabricator.wikimedia.org/P72757 and previous config saved to /var/cache/conftool/dbconfig/20250129-143448-marostegui.json
* 14:34 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2031.codfw.wmnet with OS bookworm
* 14:34 hnowlan@deploy2002: hnowlan: Continuing with sync
* 14:33 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2178.codfw.wmnet
* 14:33 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1185.eqiad.wmnet
* 14:33 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1185 db2178 T384994', diff saved to https://phabricator.wikimedia.org/P72756 and previous config saved to /var/cache/conftool/dbconfig/20250129-143317-marostegui.json
* 14:32 hnowlan@deploy2002: hnowlan: Backport for gerrit:1114398fc-list: update font list (T280718) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:28 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 14:28 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 14:27 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 14:27 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 14:25 hnowlan@deploy2002: Started scap sync-world: Backport for gerrit:1114398fc-list: update font list (T280718)
* 14:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P72755 and previous config saved to /var/cache/conftool/dbconfig/20250129-141941-marostegui.json
* 14:19 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1114727Enable mul language code on Wikidata (full release) (T312176) (duration: 16m 58s)
* 14:18 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 14:17 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 14:14 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2030.codfw.wmnet
* 14:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2031.codfw.wmnet with reason: host reimage
* 14:11 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2030.codfw.wmnet
* 14:09 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 14:09 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2031.codfw.wmnet with reason: host reimage
* 14:09 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 14:08 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde: Continuing with sync
* 14:07 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde: Backport for gerrit:1114727Enable mul language code on Wikidata (full release) (T312176) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:07 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2030.codfw.wmnet
* 14:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2029.codfw.wmnet
* 14:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P72754 and previous config saved to /var/cache/conftool/dbconfig/20250129-140434-marostegui.json
* 14:04 marostegui@cumin1002: dbctl commit (dc=all): 'db2211 (re)pooling @ 100%: Repooling after rebuild index $TASKID', diff saved to https://phabricator.wikimedia.org/P72753 and previous config saved to /var/cache/conftool/dbconfig/20250129-140418-root.json
* 14:02 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2029.codfw.wmnet
* 14:02 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2029.codfw.wmnet
* 14:02 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1114727Enable mul language code on Wikidata (full release) (T312176)
* 14:00 marostegui@cumin1002: dbctl commit (dc=all): 'db1200 (re)pooling @ 100%: Repooling after rebuild index $TASKID', diff saved to https://phabricator.wikimedia.org/P72752 and previous config saved to /var/cache/conftool/dbconfig/20250129-140052-root.json
* 13:58 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2029.codfw.wmnet
* 13:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T384592)', diff saved to https://phabricator.wikimedia.org/P72751 and previous config saved to /var/cache/conftool/dbconfig/20250129-134927-marostegui.json
* 13:49 marostegui@cumin1002: dbctl commit (dc=all): 'db2211 (re)pooling @ 75%: Repooling after rebuild index $TASKID', diff saved to https://phabricator.wikimedia.org/P72750 and previous config saved to /var/cache/conftool/dbconfig/20250129-134912-root.json
* 13:45 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2031.codfw.wmnet with OS bookworm
* 13:45 marostegui@cumin1002: dbctl commit (dc=all): 'db1200 (re)pooling @ 75%: Repooling after rebuild index $TASKID', diff saved to https://phabricator.wikimedia.org/P72749 and previous config saved to /var/cache/conftool/dbconfig/20250129-134547-root.json
* 13:42 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti2031.codfw.wmnet with OS bookworm
* 
* 13:34 marostegui@cumin1002: dbctl commit (dc=all): 'db2211 (re)pooling @ 50%: Repooling after rebuild index $TASKID', diff saved to https://phabricator.wikimedia.org/P72748 and previous config saved to /var/cache/conftool/dbconfig/20250129-133408-root.json
* 13:30 marostegui@cumin1002: dbctl commit (dc=all): 'db1200 (re)pooling @ 50%: Repooling after rebuild index $TASKID', diff saved to https://phabricator.wikimedia.org/P72747 and previous config saved to /var/cache/conftool/dbconfig/20250129-133042-root.json
* 13:27 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2031.codfw.wmnet with OS bookworm
* 13:23 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1157 (T384592)', diff saved to https://phabricator.wikimedia.org/P72746 and previous config saved to /var/cache/conftool/dbconfig/20250129-132354-marostegui.json
* 13:23 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 13:19 marostegui@cumin1002: dbctl commit (dc=all): 'db2211 (re)pooling @ 25%: Repooling after rebuild index $TASKID', diff saved to https://phabricator.wikimedia.org/P72745 and previous config saved to /var/cache/conftool/dbconfig/20250129-131903-root.json
* 13:15 marostegui@cumin1002: dbctl commit (dc=all): 'db1200 (re)pooling @ 25%: Repooling after rebuild index $TASKID', diff saved to https://phabricator.wikimedia.org/P72744 and previous config saved to /var/cache/conftool/dbconfig/20250129-131537-root.json
* 13:13 moritzm: installing runc security updates
* 13:03 marostegui@cumin1002: dbctl commit (dc=all): 'db2211 (re)pooling @ 10%: Repooling after rebuild index $TASKID', diff saved to https://phabricator.wikimedia.org/P72743 and previous config saved to /var/cache/conftool/dbconfig/20250129-130358-root.json
* 13:02 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2028.codfw.wmnet to cluster codfw and group A
* 13:01 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2028.codfw.wmnet to cluster codfw and group A
* 13:00 marostegui@cumin1002: dbctl commit (dc=all): 'db1200 (re)pooling @ 10%: Repooling after rebuild index $TASKID', diff saved to https://phabricator.wikimedia.org/P72742 and previous config saved to /var/cache/conftool/dbconfig/20250129-130031-root.json
* 12:58 aokoth@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 12:57 aokoth@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 12:56 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1212.eqiad.wmnet with reason: Index rebuild
* 12:56 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1212.eqiad.wmnet
* 12:54 aokoth@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 12:54 aokoth@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 12:50 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1212.eqiad.wmnet
* 12:50 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1013,1017].eqiad.wmnet with reason: maintenance
* 
* 12:50 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1212 T384807', diff saved to https://phabricator.wikimedia.org/P72741 and previous config saved to /var/cache/conftool/dbconfig/20250129-125015-marostegui.json
* 12:49 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db[1154,1212].eqiad.wmnet with reason: maintenance
* 12:47 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on dbstore1009.eqiad.wmnet with reason: maintenance
* 12:47 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on dbstore1007.eqiad.wmnet with reason: maintenance
* 12:37 aokoth@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 12:37 aokoth@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 12:37 aokoth@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 12:37 aokoth@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 12:35 aokoth@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 12:35 aokoth@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 12:34 marostegui: Rebuild tables on dbstore1009 (s6 s8) T384818
* 12:33 marostegui: Rebuild tables on dbstore1007 (s2, s3, s4) T384818
* 12:27 aokoth@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 12:27 aokoth@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 12:20 aokoth@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 12:20 aokoth@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 12:09 fceratto@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es1025.eqiad.wmnet
* 12:08 fceratto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:08 fceratto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es1025.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1002"
* 12:08 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2211.codfw.wmnet with reason: Index rebuild
* 12:07 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2211.codfw.wmnet
* 12:07 mvolz@deploy2002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 12:06 mvolz@deploy2002: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 12:04 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1200.eqiad.wmnet with reason: Index rebuild
* 12:04 mvolz@deploy2002: helmfile [codfw] DONE helmfile.d/services/citoid: apply
* 12:03 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1200.eqiad.wmnet
* 12:03 mvolz@deploy2002: helmfile [codfw] START helmfile.d/services/citoid: apply
* 12:03 fceratto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es1025.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1002"
* 12:03 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2211.codfw.wmnet
* 12:02 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2211 T384994', diff saved to https://phabricator.wikimedia.org/P72738 and previous config saved to /var/cache/conftool/dbconfig/20250129-120213-marostegui.json
* 12:00 mvolz@deploy2002: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 12:00 mvolz@deploy2002: helmfile [staging] START helmfile.d/services/citoid: apply
* 11:57 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1200.eqiad.wmnet
* 11:57 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1200', diff saved to https://phabricator.wikimedia.org/P72737 and previous config saved to /var/cache/conftool/dbconfig/20250129-115700-marostegui.json
* 11:51 fceratto@cumin1002: START - Cookbook sre.dns.netbox
* 11:49 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
* 11:42 fceratto@cumin1002: START - Cookbook sre.hosts.decommission for hosts es1025.eqiad.wmnet
* 11:35 jiji@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply
* 11:35 jiji@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply
* 11:35 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on netflow2003.codfw.wmnet with reason: disabling alerts as I'm running gnmic manually rather than with systemd
* 11:24 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2031.codfw.wmnet with reason: remove from cluster for reimage
* 11:24 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2031.codfw.wmnet
* 11:23 jiji@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
* 11:22 jiji@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
* 11:21 jiji@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
* 11:21 jiji@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
* 11:14 jiji@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
* 11:13 jiji@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-video: apply
* 11:13 jiji@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
* 11:13 jiji@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-video: apply
* 11:06 aokoth@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 11:05 aokoth@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 11:05 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2218.codfw.wmnet with reason: Maintenance
* 11:02 aokoth@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 11:02 aokoth@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 11:01 aokoth@deploy2002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 11:01 aokoth@deploy2002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 11:01 marostegui@cumin1002: dbctl commit (dc=all): 'db2223 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72736 and previous config saved to /var/cache/conftool/dbconfig/20250129-110125-root.json
* 11:01 marostegui@cumin1002: dbctl commit (dc=all): 'db1210 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72735 and previous config saved to /var/cache/conftool/dbconfig/20250129-110109-root.json
* 11:00 aokoth@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 11:00 aokoth@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:59 aokoth@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:59 aokoth@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:52 fceratto@cumin1002: dbctl commit (dc=all): 'Remove es1025 from dbctl T384912', diff saved to https://phabricator.wikimedia.org/P72734 and previous config saved to /var/cache/conftool/dbconfig/20250129-105232-fceratto.json
* 10:51 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1181.eqiad.wmnet with reason: Maintenance
* 10:46 marostegui@cumin1002: dbctl commit (dc=all): 'db2223 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72733 and previous config saved to /var/cache/conftool/dbconfig/20250129-104620-root.json
* 10:46 marostegui@cumin1002: dbctl commit (dc=all): 'db1210 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72732 and previous config saved to /var/cache/conftool/dbconfig/20250129-104604-root.json
* 10:46 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2222 (T384592)', diff saved to https://phabricator.wikimedia.org/P72731 and previous config saved to /var/cache/conftool/dbconfig/20250129-104558-marostegui.json
* 10:43 aokoth@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:43 aokoth@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:41 aokoth@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 10:41 aokoth@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 10:40 aokoth@deploy2002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 10:40 aokoth@deploy2002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 10:36 fceratto@cumin1002: dbctl commit (dc=all): 'Depool es1025 T384912', diff saved to https://phabricator.wikimedia.org/P72730 and previous config saved to /var/cache/conftool/dbconfig/20250129-103652-fceratto.json
* 10:31 marostegui@cumin1002: dbctl commit (dc=all): 'db2223 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72729 and previous config saved to /var/cache/conftool/dbconfig/20250129-103115-root.json
* 10:30 marostegui@cumin1002: dbctl commit (dc=all): 'db1210 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72728 and previous config saved to /var/cache/conftool/dbconfig/20250129-103059-root.json
* 10:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P72727 and previous config saved to /var/cache/conftool/dbconfig/20250129-103051-marostegui.json
* 10:26 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on clouddb1017.eqiad.wmnet with reason: maintenance
* 10:26 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on clouddb1013.eqiad.wmnet with reason: maintenance
* 10:25 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on clouddb1020.eqiad.wmnet with reason: maintenance
* 10:25 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on clouddb1016.eqiad.wmnet with reason: maintenance
* 10:21 marostegui: Upgrade and reboot db1154 (s1, s3, s5, s8 wikireplicas will get lag)
* 10:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: maintenance
* 10:18 moritzm: installing git-lfs security updates on bullseye
* 10:16 root@cumin1002: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db2156 gradually with 4 steps - Repooling after rebuild index T384807
* 10:16 marostegui@cumin1002: dbctl commit (dc=all): 'db2223 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72725 and previous config saved to /var/cache/conftool/dbconfig/20250129-101609-root.json
* 10:15 marostegui@cumin1002: dbctl commit (dc=all): 'db1210 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72724 and previous config saved to /var/cache/conftool/dbconfig/20250129-101553-root.json
* 10:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P72723 and previous config saved to /var/cache/conftool/dbconfig/20250129-101544-marostegui.json
* 10:08 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2028.codfw.wmnet
* 10:01 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on an-redacteddb1001.eqiad.wmnet with reason: maintenance
* 10:01 marostegui@cumin1002: dbctl commit (dc=all): 'db2223 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72721 and previous config saved to /var/cache/conftool/dbconfig/20250129-100104-root.json
* 10:00 marostegui@cumin1002: dbctl commit (dc=all): 'db1210 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72720 and previous config saved to /var/cache/conftool/dbconfig/20250129-100048-root.json
* 10:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2222 (T384592)', diff saved to https://phabricator.wikimedia.org/P72719 and previous config saved to /var/cache/conftool/dbconfig/20250129-100037-marostegui.json
* 10:00 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2028.codfw.wmnet
* 09:42 marostegui: Upgrade mariadb on an-redacteddb1001
* 09:41 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-redacteddb1001.eqiad.wmnet with reason: maintenance
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ganeti2028.codfw.wmnet
* 09:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2028.codfw.wmnet
* 09:31 root@cumin1002: START - Cookbook sre.mysql.pool db2156 gradually with 4 steps - Repooling after rebuild index T384807
* 09:29 marostegui@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72716 and previous config saved to /var/cache/conftool/dbconfig/20250129-092902-root.json
* 09:25 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2028.codfw.wmnet
* 09:13 marostegui@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72715 and previous config saved to /var/cache/conftool/dbconfig/20250129-091357-root.json
* 09:11 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2222 (T384592)', diff saved to https://phabricator.wikimedia.org/P72714 and previous config saved to /var/cache/conftool/dbconfig/20250129-091156-marostegui.json
* 09:11 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2222.codfw.wmnet with reason: Maintenance
* 09:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2221 (T384592)', diff saved to https://phabricator.wikimedia.org/P72713 and previous config saved to /var/cache/conftool/dbconfig/20250129-091134-marostegui.json
* 09:05 oblivian@deploy2002: Finished scap sync-world: Backport for gerrit:1113788DBRecordCache: handle default section (T382947) (duration: 10m 39s)
* 08:58 marostegui@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72711 and previous config saved to /var/cache/conftool/dbconfig/20250129-085852-root.json
* 08:58 oblivian@deploy2002: oblivian: Continuing with sync
* 08:57 oblivian@deploy2002: oblivian: Backport for gerrit:1113788DBRecordCache: handle default section (T382947) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 08:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P72710 and previous config saved to /var/cache/conftool/dbconfig/20250129-085627-marostegui.json
* 08:54 oblivian@deploy2002: Started scap sync-world: Backport for gerrit:1113788DBRecordCache: handle default section (T382947)
* 08:52 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2223.codfw.wmnet with reason: Index rebuild
* 08:51 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2223.codfw.wmnet
* 08:51 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1210.eqiad.wmnet with reason: Index rebuild
* 08:51 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1210.eqiad.wmnet
* 08:46 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1210.eqiad.wmnet
* 08:46 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2223.codfw.wmnet
* 08:46 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1210 db2223 T384994', diff saved to https://phabricator.wikimedia.org/P72707 and previous config saved to /var/cache/conftool/dbconfig/20250129-084611-marostegui.json
* 08:44 root@cumin1002: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db2157 gradually with 4 steps - Repooling after rebuild index T384994
* 08:43 marostegui@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72705 and previous config saved to /var/cache/conftool/dbconfig/20250129-084347-root.json
* 08:42 jmm@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ganeti2028.codfw.wmnet
* 08:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P72704 and previous config saved to /var/cache/conftool/dbconfig/20250129-084120-marostegui.json
* 08:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2028.codfw.wmnet with OS bookworm
* 08:31 vgutierrez: depooled lvs4009 during 60s to test lvs4010 running liberica - T384477
* 08:28 marostegui@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72702 and previous config saved to /var/cache/conftool/dbconfig/20250129-082841-root.json
* 08:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2221 (T384592)', diff saved to https://phabricator.wikimedia.org/P72701 and previous config saved to /var/cache/conftool/dbconfig/20250129-082613-marostegui.json
* 08:26 marostegui@cumin1002: dbctl commit (dc=all): 'Pool db1230', diff saved to https://phabricator.wikimedia.org/P72700 and previous config saved to /var/cache/conftool/dbconfig/20250129-082606-marostegui.json
* 08:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2028.codfw.wmnet with reason: host reimage
* 08:14 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2031.codfw.wmnet
* 08:12 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2028.codfw.wmnet with reason: host reimage
* 08:08 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2031.codfw.wmnet
* 07:58 root@cumin1002: START - Cookbook sre.mysql.pool db2157 gradually with 4 steps - Repooling after rebuild index T384994
* 07:58 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2031.codfw.wmnet
* 07:56 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2028.codfw.wmnet with OS bookworm
* 07:55 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ganeti2028.codfw.wmnet with OS bookworm
* 07:44 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp1004.wikimedia.org
* 07:42 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2028.codfw.wmnet with OS bookworm
* 07:40 root@cumin1002: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) db1230 gradually with 4 steps - Repooling after rebuild index T384994
* 07:40 root@cumin1002: START - Cookbook sre.mysql.pool db1230 gradually with 4 steps - Repooling after rebuild index T384994
* 07:40 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host idp1004.wikimedia.org
* 07:34 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2028.codfw.wmnet with reason: remove from cluster for reimage
* 07:33 moritzm: installing Tomcat security updates
* 07:16 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2221 (T384592)', diff saved to https://phabricator.wikimedia.org/P72697 and previous config saved to /var/cache/conftool/dbconfig/20250129-071632-marostegui.json
* 07:16 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2221.codfw.wmnet with reason: Maintenance
* 07:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2220 (T384592)', diff saved to https://phabricator.wikimedia.org/P72696 and previous config saved to /var/cache/conftool/dbconfig/20250129-071610-marostegui.json
* 07:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P72695 and previous config saved to /var/cache/conftool/dbconfig/20250129-070103-marostegui.json
* 06:53 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2157.codfw.wmnet with reason: Index rebuild
* 06:52 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1230.eqiad.wmnet with reason: Index rebuild
* 06:52 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2157.codfw.wmnet
* 06:51 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1230.eqiad.wmnet
* 06:46 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2157.codfw.wmnet
* 06:46 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1230.eqiad.wmnet
* 06:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P72694 and previous config saved to /var/cache/conftool/dbconfig/20250129-064555-marostegui.json
* 06:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1230 db2157 T384994', diff saved to https://phabricator.wikimedia.org/P72693 and previous config saved to /var/cache/conftool/dbconfig/20250129-064545-marostegui.json
* 06:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2220 (T384592)', diff saved to https://phabricator.wikimedia.org/P72692 and previous config saved to /var/cache/conftool/dbconfig/20250129-063015-marostegui.json
* 06:20 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2156.codfw.wmnet with reason: Index rebuild
* 06:19 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2156.codfw.wmnet
* 06:13 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2156.codfw.wmnet
* 06:12 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2186.codfw.wmnet with reason: maintenance
* 06:12 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2156.codfw.wmnet with reason: maintenance
* 06:12 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2156', diff saved to https://phabricator.wikimedia.org/P72691 and previous config saved to /var/cache/conftool/dbconfig/20250129-061214-marostegui.json
* 05:41 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2220 (T384592)', diff saved to https://phabricator.wikimedia.org/P72690 and previous config saved to /var/cache/conftool/dbconfig/20250129-054145-marostegui.json
* 05:41 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2220.codfw.wmnet with reason: Maintenance
* 05:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2208 (T384592)', diff saved to https://phabricator.wikimedia.org/P72689 and previous config saved to /var/cache/conftool/dbconfig/20250129-054121-marostegui.json
* 05:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P72688 and previous config saved to /var/cache/conftool/dbconfig/20250129-052615-marostegui.json
* 05:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P72687 and previous config saved to /var/cache/conftool/dbconfig/20250129-051108-marostegui.json
* 05:08 ejegg: fundraising civicrm upgraded from to 
* 04:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2208 (T384592)', diff saved to https://phabricator.wikimedia.org/P72686 and previous config saved to /var/cache/conftool/dbconfig/20250129-045600-marostegui.json
* 04:08 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2208 (T384592)', diff saved to https://phabricator.wikimedia.org/P72685 and previous config saved to /var/cache/conftool/dbconfig/20250129-040822-marostegui.json
* 04:08 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2208.codfw.wmnet with reason: Maintenance
* 03:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2200.codfw.wmnet with reason: Maintenance
* 02:56 denisse@deploy2002: Finished deploy [librenms/librenms@f049593]: Upgrade LibreNMS to 25.1.0 - T384258 (duration: 00m 13s)
* 02:56 denisse@deploy2002: Started deploy [librenms/librenms@f049593]: Upgrade LibreNMS to 25.1.0 - T384258
* 01:57 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 01:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T384592)', diff saved to https://phabricator.wikimedia.org/P72683 and previous config saved to /var/cache/conftool/dbconfig/20250129-015717-marostegui.json
* 01:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P72682 and previous config saved to /var/cache/conftool/dbconfig/20250129-014210-marostegui.json
* 01:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P72681 and previous config saved to /var/cache/conftool/dbconfig/20250129-012703-marostegui.json
* 01:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T384592)', diff saved to https://phabricator.wikimedia.org/P72680 and previous config saved to /var/cache/conftool/dbconfig/20250129-011157-marostegui.json
* 00:17 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2182 (T384592)', diff saved to https://phabricator.wikimedia.org/P72679 and previous config saved to /var/cache/conftool/dbconfig/20250129-001702-marostegui.json
* 00:16 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2182.codfw.wmnet with reason: Maintenance
* 00:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2168 (T384592)', diff saved to https://phabricator.wikimedia.org/P72678 and previous config saved to /var/cache/conftool/dbconfig/20250129-001651-marostegui.json
* 00:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P72677 and previous config saved to /var/cache/conftool/dbconfig/20250129-000144-marostegui.json
== 2025-01-28 ==
* 23:46 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P72676 and previous config saved to /var/cache/conftool/dbconfig/20250128-234637-marostegui.json
* 23:31 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2168 (T384592)', diff saved to https://phabricator.wikimedia.org/P72675 and previous config saved to /var/cache/conftool/dbconfig/20250128-233130-marostegui.json
* 22:36 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2168 (T384592)', diff saved to https://phabricator.wikimedia.org/P72673 and previous config saved to /var/cache/conftool/dbconfig/20250128-223652-marostegui.json
* 22:36 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2168.codfw.wmnet with reason: Maintenance
* 22:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T384592)', diff saved to https://phabricator.wikimedia.org/P72672 and previous config saved to /var/cache/conftool/dbconfig/20250128-223630-marostegui.json
* 22:30 jhuneidi@deploy2002: Finished scap sync-world: Backport for gerrit:1114804Move VE EditCheck testwiki enabling into the correct file (T384658), gerrit:1114425Condense wikivoyage configuration options for Parsoid Read Views (T365367) (duration: 13m 48s)
* 22:23 jhuneidi@deploy2002: jhuneidi, cscott, kemayo: Continuing with sync
* 22:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P72670 and previous config saved to /var/cache/conftool/dbconfig/20250128-222123-marostegui.json
* 22:06 jhuneidi@deploy2002: jhuneidi, matmarex: Continuing with sync
* 22:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P72667 and previous config saved to /var/cache/conftool/dbconfig/20250128-220616-marostegui.json
* 22:06 jhuneidi@deploy2002: jhuneidi, matmarex: Backport for gerrit:1114799Fix PHP 7.4 issue (T384905), [[gerrit:1114800wikimedia/request-timeout: 2.0.1 -> 2.0.2 (T384905)]], [[gerrit:1114802composer: wikimedia/request-timeout 2.0.1 -> 2.0.2 (T384905)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 22:03 aqu@deploy2002: Finished deploy [analytics/refinery@3959b36] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@3959b36b] (duration: 00m 34s)
* 22:02 aqu@deploy2002: Started deploy [analytics/refinery@3959b36] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@3959b36b]
* 22:02 aqu@deploy2002: Finished deploy [analytics/refinery@3959b36] (thin): Regular analytics weekly train THIN [analytics/refinery@3959b36b] (duration: 01m 08s)
* 22:01 jhuneidi@deploy2002: Started scap sync-world: Backport for gerrit:1114799Fix PHP 7.4 issue (T384905), [[gerrit:1114800wikimedia/request-timeout: 2.0.1 -> 2.0.2 (T384905)]], [[gerrit:1114802composer: wikimedia/request-timeout 2.0.1 -> 2.0.2 (T384905)]]
* 22:00 aqu@deploy2002: Started deploy [analytics/refinery@3959b36] (thin): Regular analytics weekly train THIN [analytics/refinery@3959b36b]
* 22:00 aqu@deploy2002: Finished deploy [analytics/refinery@3959b36]: Regular analytics weekly train [analytics/refinery@3959b36b] (duration: 02m 03s)
* 21:58 aqu@deploy2002: Started deploy [analytics/refinery@3959b36]: Regular analytics weekly train [analytics/refinery@3959b36b]
* 21:57 aqu: About to deploy analytics/refinery
* 21:56 aqu: Deployed refinery-source using jenkins
* 21:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T384592)', diff saved to https://phabricator.wikimedia.org/P72666 and previous config saved to /var/cache/conftool/dbconfig/20250128-215109-marostegui.json
* 20:49 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2159 (T384592)', diff saved to https://phabricator.wikimedia.org/P72665 and previous config saved to /var/cache/conftool/dbconfig/20250128-204933-marostegui.json
* 20:49 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2187.codfw.wmnet with reason: Maintenance
* 20:49 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2159.codfw.wmnet with reason: Maintenance
* 20:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T384592)', diff saved to https://phabricator.wikimedia.org/P72664 and previous config saved to /var/cache/conftool/dbconfig/20250128-204907-marostegui.json
* 20:47 marostegui@cumin1002: dbctl commit (dc=all): 'db2177 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72663 and previous config saved to /var/cache/conftool/dbconfig/20250128-204712-root.json
* 20:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P72662 and previous config saved to /var/cache/conftool/dbconfig/20250128-203400-marostegui.json
* 20:32 marostegui@cumin1002: dbctl commit (dc=all): 'db2177 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72661 and previous config saved to /var/cache/conftool/dbconfig/20250128-203207-root.json
* 20:18 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P72660 and previous config saved to /var/cache/conftool/dbconfig/20250128-201853-marostegui.json
* 20:17 marostegui@cumin1002: dbctl commit (dc=all): 'db2177 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72659 and previous config saved to /var/cache/conftool/dbconfig/20250128-201702-root.json
* 20:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T384592)', diff saved to https://phabricator.wikimedia.org/P72656 and previous config saved to /var/cache/conftool/dbconfig/20250128-200346-marostegui.json
* 20:01 marostegui@cumin1002: dbctl commit (dc=all): 'db2177 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72655 and previous config saved to /var/cache/conftool/dbconfig/20250128-200157-root.json
* 19:46 marostegui@cumin1002: dbctl commit (dc=all): 'db2177 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72654 and previous config saved to /var/cache/conftool/dbconfig/20250128-194651-root.json
* 19:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2028.codfw.wmnet
* 19:20 jhuneidi@deploy2002: rebuilt and synchronized wikiversions files: group0 to 1.44.0-wmf.14 refs T382365
* 18:21 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
* 18:18 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-video: apply
* 18:18 swfrench-wmf: starting shellbox-video pilot on PHP 8.1 (3 replicas, codfw only) - T377038
* 18:17 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2150 (T384592)', diff saved to https://phabricator.wikimedia.org/P72653 and previous config saved to /var/cache/conftool/dbconfig/20250128-181729-marostegui.json
* 18:17 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2150.codfw.wmnet with reason: Maintenance
* 18:05 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
* 18:04 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
* 18:04 swfrench-wmf: starting shellbox-constraints pilot on PHP 8.1 (1 replica, eqiad only) - T377038
* 18:03 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
* 18:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236 (T384592)', diff saved to https://phabricator.wikimedia.org/P72652 and previous config saved to /var/cache/conftool/dbconfig/20250128-180335-marostegui.json
* 17:48 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P72651 and previous config saved to /var/cache/conftool/dbconfig/20250128-174828-marostegui.json
* 17:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P72650 and previous config saved to /var/cache/conftool/dbconfig/20250128-173321-marostegui.json
* 17:18 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236 (T384592)', diff saved to https://phabricator.wikimedia.org/P72649 and previous config saved to /var/cache/conftool/dbconfig/20250128-171814-marostegui.json
* 17:12 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72648 and previous config saved to /var/cache/conftool/dbconfig/20250128-171205-root.json
* 17:10 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs4010.ulsfo.wmnet with OS bookworm
* 17:06 rzl: stopping puppet on A:cp-text
* 17:05 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on netflow3003.esams.wmnet with reason: disabling alerts as I'm running gnmic manually rather than with systemd
* 17:04 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1236 (T384592)', diff saved to https://phabricator.wikimedia.org/P72647 and previous config saved to /var/cache/conftool/dbconfig/20250128-170412-marostegui.json
* 17:04 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1236.eqiad.wmnet with reason: Maintenance
* 17:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227 (T384592)', diff saved to https://phabricator.wikimedia.org/P72646 and previous config saved to /var/cache/conftool/dbconfig/20250128-170350-marostegui.json
* 17:01 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr[1-2]-magru,cr[1-2]-magru IPv6
* 17:01 cmooney@cumin1002: START - Cookbook sre.hosts.remove-downtime for cr[1-2]-magru,cr[1-2]-magru IPv6
* 16:57 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72645 and previous config saved to /var/cache/conftool/dbconfig/20250128-165700-root.json
* 16:53 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 16:53 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 16:52 elukey: restart kartotherian on maps1009 as test
* 16:51 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 16:51 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 16:48 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P72644 and previous config saved to /var/cache/conftool/dbconfig/20250128-164843-marostegui.json
* 16:42 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 16:42 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 16:42 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 16:42 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 16:41 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72643 and previous config saved to /var/cache/conftool/dbconfig/20250128-164154-root.json
* 16:41 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 16:41 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 16:41 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 16:41 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 16:40 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 16:40 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 16:37 vgutierrez@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs4010.ulsfo.wmnet with reason: host reimage
* 16:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P72642 and previous config saved to /var/cache/conftool/dbconfig/20250128-163336-marostegui.json
* 16:33 vgutierrez@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs4010.ulsfo.wmnet with reason: host reimage
* 16:26 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2177.codfw.wmnet with reason: Index rebuild
* 16:26 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72641 and previous config saved to /var/cache/conftool/dbconfig/20250128-162649-root.json
* 16:26 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2177.codfw.wmnet
* 16:25 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host gerrit2002
* 16:25 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host gerrit2002
* 16:25 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2177.codfw.wmnet with reason: maintenance
* 16:21 reedy@deploy2002: Finished scap sync-world: Backport for gerrit:1114701FormatMetadata: Prevent running preg_match() on null (T384879), gerrit:1114702FormatMetadata: Prevent running preg_match() on null (T384879) (duration: 12m 12s)
* 16:19 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2177.codfw.wmnet
* 16:18 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2177 T382842', diff saved to https://phabricator.wikimedia.org/P72640 and previous config saved to /var/cache/conftool/dbconfig/20250128-161857-marostegui.json
* 16:18 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227 (T384592)', diff saved to https://phabricator.wikimedia.org/P72639 and previous config saved to /var/cache/conftool/dbconfig/20250128-161829-marostegui.json
* 16:15 cgoubert@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
* 16:15 cgoubert@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
* 16:15 vgutierrez@cumin1002: START - Cookbook sre.hosts.reimage for host lvs4010.ulsfo.wmnet with OS bookworm
* 16:14 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2013,2036,2088].codfw.wmnet
* 16:14 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker[2013,2036,2088].codfw.wmnet
* 16:14 jayme@cumin1002: START - Cookbook sre.hosts.remove-downtime for wikikube-worker[2013,2036,2088].codfw.wmnet
* 16:14 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2013,2036,2088].codfw.wmnet
* 16:14 reedy@deploy2002: reedy: Continuing with sync
* 16:13 reedy@deploy2002: reedy: Backport for gerrit:1114701FormatMetadata: Prevent running preg_match() on null (T384879), gerrit:1114702FormatMetadata: Prevent running preg_match() on null (T384879) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 16:11 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2088
* 16:11 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72638 and previous config saved to /var/cache/conftool/dbconfig/20250128-161143-root.json
* 16:11 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2088
* 16:08 reedy@deploy2002: Started scap sync-world: Backport for gerrit:1114701FormatMetadata: Prevent running preg_match() on null (T384879), gerrit:1114702FormatMetadata: Prevent running preg_match() on null (T384879)
* 16:06 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet
* 16:06 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-ctrl1003.eqiad.wmnet
* 16:06 jayme@cumin1002: START - Cookbook sre.hosts.remove-downtime for wikikube-ctrl1003.eqiad.wmnet
* 16:06 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet
* 16:05 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1166', diff saved to https://phabricator.wikimedia.org/P72637 and previous config saved to /var/cache/conftool/dbconfig/20250128-160518-marostegui.json
* 16:04 root@cumin1002: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) db1166 gradually with 4 steps - Repooling after rebuild index T384807
* 16:03 root@cumin1002: START - Cookbook sre.mysql.pool db1166 gradually with 4 steps - Repooling after rebuild index T384807
* 16:00 jelto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gerrit2002.wikimedia.org with reason: NIC port switch -t T383709
* 15:59 cgoubert@deploy2002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 15:59 cgoubert@deploy2002: helmfile [codfw] START helmfile.d/admin 'apply'.
* 15:58 vgutierrez: upload liberica 0.7 to apt.wm.o (bookworm-wikimedia)
* 15:57 cgoubert@deploy2002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 15:57 cgoubert@deploy2002: helmfile [codfw] START helmfile.d/admin 'apply'.
* 15:56 cgoubert@deploy2002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 15:56 cgoubert@deploy2002: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 15:55 cgoubert@deploy2002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 15:54 cgoubert@deploy2002: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 15:54 cgoubert@deploy2002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 15:53 cgoubert@deploy2002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 15:52 cgoubert@deploy2002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 15:51 cgoubert@deploy2002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 15:50 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2036
* 15:50 cgoubert@deploy2002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 15:50 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2036
* 15:49 cgoubert@deploy2002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 15:49 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet
* 15:49 jayme@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl1003.eqiad.wmnet with reason: Depooled via sre.k8s.pool-depool-node
* 15:49 cgoubert@deploy2002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 15:49 cgoubert@deploy2002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 15:49 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet
* 15:48 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet
* 15:48 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-ctrl1002.eqiad.wmnet
* 15:48 jayme@cumin1002: START - Cookbook sre.hosts.remove-downtime for wikikube-ctrl1002.eqiad.wmnet
* 15:48 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet
* 15:48 cgoubert@deploy2002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 15:48 aqu: About to deploy analytics/refinery/source 0.2.57
* 15:48 cgoubert@deploy2002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 15:47 cgoubert@deploy2002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 15:47 cgoubert@deploy2002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 15:46 cgoubert@deploy2002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 15:45 cgoubert@deploy2002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 15:38 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2013
* 15:38 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2013
* 15:22 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1227 (T384592)', diff saved to https://phabricator.wikimedia.org/P72635 and previous config saved to /var/cache/conftool/dbconfig/20250128-152159-marostegui.json
* 15:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1227.eqiad.wmnet with reason: Maintenance
* 15:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T384592)', diff saved to https://phabricator.wikimedia.org/P72634 and previous config saved to /var/cache/conftool/dbconfig/20250128-152137-marostegui.json
* 15:14 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2028.codfw.wmnet
* 15:13 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd2003.codfw.wmnet to plain
* 15:12 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd2003.codfw.wmnet to plain
* 15:11 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2028.codfw.wmnet
* 15:10 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2028.codfw.wmnet
* 15:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P72631 and previous config saved to /var/cache/conftool/dbconfig/20250128-150630-marostegui.json
* 15:06 jelto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gerrit2002.wikimedia.org with reason: NIC port switch -t T383709
* 15:05 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd2003.codfw.wmnet to drbd
* 14:57 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2013,2036,2088].codfw.wmnet
* 14:55 jayme@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on wikikube-worker[2013,2036,2088].codfw.wmnet with reason: Depooled via sre.k8s.pool-depool-node
* 14:55 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2013,2036,2088].codfw.wmnet
* 14:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P72630 and previous config saved to /var/cache/conftool/dbconfig/20250128-145123-marostegui.json
* 14:48 oblivian@deploy2002: Finished scap sync-world: Backport for gerrit:1114703zhwiki: Add 2025 CNY celebration logos (T384913) (duration: 15m 40s)
* 14:44 klausman@deploy2002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 14:43 klausman@deploy2002: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 14:42 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd2003.codfw.wmnet to drbd
* 14:41 oblivian@deploy2002: stang, oblivian: Continuing with sync
* 14:40 vgutierrez: updating to liberica 0.6 in lvs1013
* 14:39 vgutierrez: upload liberica 0.6 to apt.wm.o (bookworm-wikimedia)
* 14:37 oblivian@deploy2002: stang, oblivian: Backport for gerrit:1114703zhwiki: Add 2025 CNY celebration logos (T384913) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T384592)', diff saved to https://phabricator.wikimedia.org/P72629 and previous config saved to /var/cache/conftool/dbconfig/20250128-143616-marostegui.json
* 14:34 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2028.codfw.wmnet
* 14:33 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet
* 14:33 jayme@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl1002.eqiad.wmnet with reason: Depooled via sre.k8s.pool-depool-node
* 14:33 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet
* 14:33 oblivian@deploy2002: Started scap sync-world: Backport for gerrit:1114703zhwiki: Add 2025 CNY celebration logos (T384913)
* 14:30 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2028.codfw.wmnet
* 14:29 jelto: Imported helm311 3.11.3-3 to bullseye-wikimedia - T341984
* 14:28 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 14:28 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 14:28 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 14:27 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 14:25 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1114440prod: Enable $wgCampaignEventsEnableEventTopics (T380818) (duration: 21m 04s)
* 14:22 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1202 (T384592)', diff saved to https://phabricator.wikimedia.org/P72628 and previous config saved to /var/cache/conftool/dbconfig/20250128-142244-marostegui.json
* 14:22 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1202.eqiad.wmnet with reason: Maintenance
* 14:22 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T384592)', diff saved to https://phabricator.wikimedia.org/P72627 and previous config saved to /var/cache/conftool/dbconfig/20250128-142222-marostegui.json
* 14:21 jelto: Imported helm311 3.11.3-3 to bookworm-wikimedia - T341984
* 14:18 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, daimona: Continuing with sync
* 14:18 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 14:18 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 14:11 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2026.codfw.wmnet to cluster codfw and group D
* 14:09 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2026.codfw.wmnet to cluster codfw and group D
* 14:09 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, daimona: Backport for gerrit:1114440prod: Enable $wgCampaignEventsEnableEventTopics (T380818) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P72626 and previous config saved to /var/cache/conftool/dbconfig/20250128-140715-marostegui.json
* 14:06 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2026.codfw.wmnet
* 14:04 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1114440prod: Enable $wgCampaignEventsEnableEventTopics (T380818)
* 13:58 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2026.codfw.wmnet
* 13:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P72625 and previous config saved to /var/cache/conftool/dbconfig/20250128-135208-marostegui.json
* 13:51 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 13:50 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 13:49 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2026.codfw.wmnet with OS bookworm
* 13:39 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 13:39 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 13:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T384592)', diff saved to https://phabricator.wikimedia.org/P72624 and previous config saved to /var/cache/conftool/dbconfig/20250128-133701-marostegui.json
* 13:33 fabfur: installing/enabling haproxykafka on eqiad (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1114417) (T378578)
* 13:27 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1166.eqiad.wmnet with reason: Index rebuild
* 13:27 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2026.codfw.wmnet with reason: host reimage
* 13:26 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1166.eqiad.wmnet
* 13:23 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2026.codfw.wmnet with reason: host reimage
* 13:22 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1194 (T384592)', diff saved to https://phabricator.wikimedia.org/P72623 and previous config saved to /var/cache/conftool/dbconfig/20250128-132238-marostegui.json
* 13:22 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1194.eqiad.wmnet with reason: Maintenance
* 13:22 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T384592)', diff saved to https://phabricator.wikimedia.org/P72622 and previous config saved to /var/cache/conftool/dbconfig/20250128-132227-marostegui.json
* 13:22 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 13:20 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 13:19 fceratto@dns1004: END - running authdns-update
* 13:19 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1166.eqiad.wmnet
* 13:18 root@cumin1002: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db2190 gradually with 4 steps - Repooling after rebuild index
* 13:17 fceratto@dns1004: START - running authdns-update
* 13:15 dbrant@deploy2002: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 13:15 dbrant@deploy2002: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 13:14 dbrant@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 13:13 dbrant@deploy2002: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 13:13 dbrant@deploy2002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 13:12 dbrant@deploy2002: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 13:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P72619 and previous config saved to /var/cache/conftool/dbconfig/20250128-130720-marostegui.json
* 13:07 dbrant@deploy2002: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 13:06 dbrant@deploy2002: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 13:06 dbrant@deploy2002: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 13:05 dbrant@deploy2002: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 13:04 dbrant@deploy2002: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 13:03 dbrant@deploy2002: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 13:03 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2026.codfw.wmnet with OS bookworm
* 13:02 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti2026.codfw.wmnet with OS bookworm
* 12:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P72617 and previous config saved to /var/cache/conftool/dbconfig/20250128-125213-marostegui.json
* 12:51 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on netflow3003.esams.wmnet with reason: disabling alerts as I'm running gnmic manually rather than with systemd
* 12:50 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudgw1003.eqiad.wmnet with OS bookworm
* 12:50 andrew@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - andrew@cumin1002"
* 12:50 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudgw1004.eqiad.wmnet with OS bookworm
* 12:50 andrew@cumin1002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - andrew@cumin1002"
* 12:45 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2026.codfw.wmnet with OS bookworm
* 12:41 andrew@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - andrew@cumin1002"
* 12:39 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1166.eqiad.wmnet with reason: Index rebuild
* 12:37 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1166 T382842', diff saved to https://phabricator.wikimedia.org/P72615 and previous config saved to /var/cache/conftool/dbconfig/20250128-123713-marostegui.json
* 12:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T384592)', diff saved to https://phabricator.wikimedia.org/P72614 and previous config saved to /var/cache/conftool/dbconfig/20250128-123706-marostegui.json
* 12:32 andrew@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - andrew@cumin1002"
* 12:32 root@cumin1002: START - Cookbook sre.mysql.pool db2190 gradually with 4 steps - Repooling after rebuild index
* 12:31 elukey@deploy2002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
* 12:30 elukey@deploy2002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
* 12:27 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2230.codfw.wmnet with reason: Index rebuild
* 12:27 slyngshede@dns1004: END - running authdns-update
* 12:27 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2230.codfw.wmnet with reason: Index rebuild
* 12:25 slyngshede@dns1004: START - running authdns-update
* 12:24 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1191 (T384592)', diff saved to https://phabricator.wikimedia.org/P72611 and previous config saved to /var/cache/conftool/dbconfig/20250128-122428-marostegui.json
* 12:24 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1191.eqiad.wmnet with reason: Maintenance
* 12:24 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T384592)', diff saved to https://phabricator.wikimedia.org/P72610 and previous config saved to /var/cache/conftool/dbconfig/20250128-122406-marostegui.json
* 12:23 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw1004.eqiad.wmnet with reason: host reimage
* 12:22 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on netflow2003.codfw.wmnet with reason: disabling alerts as I'm running gnmic manually rather than with systemd
* 12:19 andrew@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw1004.eqiad.wmnet with reason: host reimage
* 12:12 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw1003.eqiad.wmnet with reason: host reimage
* 12:09 andrew@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw1003.eqiad.wmnet with reason: host reimage
* 12:09 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P72609 and previous config saved to /var/cache/conftool/dbconfig/20250128-120859-marostegui.json
* 12:02 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudgw1004.eqiad.wmnet with OS bookworm
* 11:56 slyngshede@dns1004: END - running authdns-update
* 11:54 slyngshede@dns1004: START - running authdns-update
* 11:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P72608 and previous config saved to /var/cache/conftool/dbconfig/20250128-115352-marostegui.json
* 11:52 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudgw1003.eqiad.wmnet with OS bookworm
* 11:45 marostegui@cumin1002: dbctl commit (dc=all): 'es2030 (re)pooling @ 100%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72607 and previous config saved to /var/cache/conftool/dbconfig/20250128-114458-root.json
* 11:44 marostegui@cumin1002: dbctl commit (dc=all): 'db2155 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72606 and previous config saved to /var/cache/conftool/dbconfig/20250128-114436-root.json
* 11:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T384592)', diff saved to https://phabricator.wikimedia.org/P72605 and previous config saved to /var/cache/conftool/dbconfig/20250128-113845-marostegui.json
* 11:35 volans: installed spicerack v9.1.1 on cumin1002
* 11:34 volans: installed spicerack v9.1.1 on cumin2002
* 11:29 marostegui@cumin1002: dbctl commit (dc=all): 'es2030 (re)pooling @ 75%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72604 and previous config saved to /var/cache/conftool/dbconfig/20250128-112952-root.json
* 11:29 marostegui@cumin1002: dbctl commit (dc=all): 'db2155 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72603 and previous config saved to /var/cache/conftool/dbconfig/20250128-112931-root.json
* 11:26 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1174 (T384592)', diff saved to https://phabricator.wikimedia.org/P72602 and previous config saved to /var/cache/conftool/dbconfig/20250128-112631-marostegui.json
* 11:26 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 11:19 klausman@deploy2002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 11:19 volans: uploaded spicerack_9.1.1 to apt.wikimedia.org bullseye-wikimedia
* 11:18 jiji@deploy2002: Finished scap sync-world: Backport for gerrit:1114657Enroll 2% of client sessions in PHP 8.1 (T383845) (duration: 11m 48s)
* 11:17 klausman@deploy2002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 11:14 marostegui@cumin1002: dbctl commit (dc=all): 'es2030 (re)pooling @ 50%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72601 and previous config saved to /var/cache/conftool/dbconfig/20250128-111447-root.json
* 11:14 marostegui@cumin1002: dbctl commit (dc=all): 'db2155 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72600 and previous config saved to /var/cache/conftool/dbconfig/20250128-111425-root.json
* 11:11 jiji@deploy2002: jiji: Continuing with sync
* 11:11 jiji@deploy2002: jiji: Backport for gerrit:1114657Enroll 2% of client sessions in PHP 8.1 (T383845) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 11:06 jiji@deploy2002: Started scap sync-world: Backport for gerrit:1114657Enroll 2% of client sessions in PHP 8.1 (T383845)
* 11:04 moritzm: installing runc security updates
* 11:03 klausman@deploy2002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 11:02 klausman@deploy2002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 11:02 klausman@deploy2002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 11:01 klausman@deploy2002: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:59 marostegui@cumin1002: dbctl commit (dc=all): 'es2030 (re)pooling @ 25%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72599 and previous config saved to /var/cache/conftool/dbconfig/20250128-105942-root.json
* 10:59 marostegui@cumin1002: dbctl commit (dc=all): 'db2155 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72598 and previous config saved to /var/cache/conftool/dbconfig/20250128-105920-root.json
* 10:54 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2020.codfw.wmnet to cluster codfw and group B
* 10:54 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2020.codfw.wmnet to cluster codfw and group B
* 10:44 marostegui@cumin1002: dbctl commit (dc=all): 'es2030 (re)pooling @ 10%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72597 and previous config saved to /var/cache/conftool/dbconfig/20250128-104436-root.json
* 10:44 marostegui@cumin1002: dbctl commit (dc=all): 'db2155 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72596 and previous config saved to /var/cache/conftool/dbconfig/20250128-104415-root.json
* 10:38 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2026.codfw.wmnet with reason: remove from cluster for reimage
* 10:36 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2026.codfw.wmnet
* 10:36 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
* 10:35 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170 (T384592)', diff saved to https://phabricator.wikimedia.org/P72595 and previous config saved to /var/cache/conftool/dbconfig/20250128-103553-marostegui.json
* 10:35 vgutierrez@cumin1002: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool site magru [reason: no reason specified, T384774]
* 10:35 vgutierrez@cumin1002: START - Cookbook sre.dns.admin DNS admin: pool site magru [reason: no reason specified, T384774]
* 10:34 marostegui@cumin1002: dbctl commit (dc=all): 'db2205 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72594 and previous config saved to /var/cache/conftool/dbconfig/20250128-103444-root.json
* 10:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P72593 and previous config saved to /var/cache/conftool/dbconfig/20250128-102046-marostegui.json
* 10:19 marostegui@cumin1002: dbctl commit (dc=all): 'db2205 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72592 and previous config saved to /var/cache/conftool/dbconfig/20250128-101939-root.json
* 10:19 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 100%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72591 and previous config saved to /var/cache/conftool/dbconfig/20250128-101908-root.json
* 10:13 marostegui@cumin1002: dbctl commit (dc=all): 'es2036 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72590 and previous config saved to /var/cache/conftool/dbconfig/20250128-101324-root.json
* 10:12 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2155 T384807', diff saved to https://phabricator.wikimedia.org/P72589 and previous config saved to /var/cache/conftool/dbconfig/20250128-101224-marostegui.json
* 10:07 marostegui@cumin1002: dbctl commit (dc=all): 'db2155 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P72588 and previous config saved to /var/cache/conftool/dbconfig/20250128-100754-root.json
* 10:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P72587 and previous config saved to /var/cache/conftool/dbconfig/20250128-100539-marostegui.json
* 10:04 marostegui@cumin1002: dbctl commit (dc=all): 'db2205 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72586 and previous config saved to /var/cache/conftool/dbconfig/20250128-100434-root.json
* 10:04 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 75%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72585 and previous config saved to /var/cache/conftool/dbconfig/20250128-100402-root.json
* 09:58 marostegui@cumin1002: dbctl commit (dc=all): 'es2036 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72584 and previous config saved to /var/cache/conftool/dbconfig/20250128-095818-root.json
* 09:50 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170 (T384592)', diff saved to https://phabricator.wikimedia.org/P72583 and previous config saved to /var/cache/conftool/dbconfig/20250128-095032-marostegui.json
* 09:50 root@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1150.eqiad.wmnet with OS bookworm
* 09:49 marostegui@cumin1002: dbctl commit (dc=all): 'db2205 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72582 and previous config saved to /var/cache/conftool/dbconfig/20250128-094928-root.json
* 09:48 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 50%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72581 and previous config saved to /var/cache/conftool/dbconfig/20250128-094857-root.json
* 09:43 marostegui@cumin1002: dbctl commit (dc=all): 'es2036 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72580 and previous config saved to /var/cache/conftool/dbconfig/20250128-094313-root.json
* 09:41 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2190.codfw.wmnet with reason: Index rebuild
* 09:39 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2190.codfw.wmnet
* 09:34 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2190.codfw.wmnet
* 09:34 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2190', diff saved to https://phabricator.wikimedia.org/P72578 and previous config saved to /var/cache/conftool/dbconfig/20250128-093446-marostegui.json
* 09:34 marostegui@cumin1002: dbctl commit (dc=all): 'db2205 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72577 and previous config saved to /var/cache/conftool/dbconfig/20250128-093423-root.json
* 09:33 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 25%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72576 and previous config saved to /var/cache/conftool/dbconfig/20250128-093352-root.json
* 09:28 marostegui@cumin1002: dbctl commit (dc=all): 'es2036 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72575 and previous config saved to /var/cache/conftool/dbconfig/20250128-092808-root.json
* 09:27 fabfur: installing/enabling haproxykafka on codfw (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1114415) (T378578)
* 09:26 root@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1150.eqiad.wmnet with reason: host reimage
* 09:22 root@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1150.eqiad.wmnet with reason: host reimage
* 09:22 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for es2030.codfw.wmnet
* 09:18 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 10%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72574 and previous config saved to /var/cache/conftool/dbconfig/20250128-091846-root.json
* 09:13 root@cumin1002: START - Cookbook sre.mysql.upgrade for es2030.codfw.wmnet
* 09:13 marostegui@cumin1002: dbctl commit (dc=all): 'es2036 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72572 and previous config saved to /var/cache/conftool/dbconfig/20250128-091302-root.json
* 09:12 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2030', diff saved to https://phabricator.wikimedia.org/P72571 and previous config saved to /var/cache/conftool/dbconfig/20250128-091242-marostegui.json
* 09:06 root@cumin1002: START - Cookbook sre.hosts.reimage for host db1150.eqiad.wmnet with OS bookworm
* 09:06 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1175', diff saved to https://phabricator.wikimedia.org/P72570 and previous config saved to /var/cache/conftool/dbconfig/20250128-090620-marostegui.json
* 09:06 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2205', diff saved to https://phabricator.wikimedia.org/P72569 and previous config saved to /var/cache/conftool/dbconfig/20250128-090601-marostegui.json
* 09:05 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for es2036.codfw.wmnet
* 09:05 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2020.codfw.wmnet
* 09:05 marostegui@cumin1002: dbctl commit (dc=all): 'db2205 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P72568 and previous config saved to /var/cache/conftool/dbconfig/20250128-090525-root.json
* 09:04 marostegui@cumin1002: dbctl commit (dc=all): 'Promote es2028 to es1 codfw master', diff saved to https://phabricator.wikimedia.org/P72567 and previous config saved to /var/cache/conftool/dbconfig/20250128-090454-marostegui.json
* 09:04 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P72566 and previous config saved to /var/cache/conftool/dbconfig/20250128-090439-root.json
* 09:00 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1150.eqiad.wmnet with reason: reimage
* 09:00 root@cumin1002: START - Cookbook sre.mysql.upgrade for es2036.codfw.wmnet
* 09:00 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2036', diff saved to https://phabricator.wikimedia.org/P72565 and previous config saved to /var/cache/conftool/dbconfig/20250128-090000-marostegui.json
* 08:57 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2020.codfw.wmnet
* 
* 08:55 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1170 (T384592)', diff saved to https://phabricator.wikimedia.org/P72564 and previous config saved to /var/cache/conftool/dbconfig/20250128-085528-marostegui.json
* 08:55 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 08:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T384592)', diff saved to https://phabricator.wikimedia.org/P72563 and previous config saved to /var/cache/conftool/dbconfig/20250128-085506-marostegui.json
* 08:51 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2020.codfw.wmnet with OS bookworm
* 08:48 reedy@deploy2002: reedy, jforrester: Continuing with sync
* 
* 
* 08:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P72562 and previous config saved to /var/cache/conftool/dbconfig/20250128-083959-marostegui.json
* 08:33 urbanecm@deploy2002: Finished scap sync-world: Backport for [[gerrit:1114379[Growth] enwiki: Release Add Link to 10% of newcomers (T384551)]] (duration: 16m 46s)
* 08:24 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P72561 and previous config saved to /var/cache/conftool/dbconfig/20250128-082452-marostegui.json
* 08:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2020.codfw.wmnet with reason: host reimage
* 08:24 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2026.codfw.wmnet
* 08:23 urbanecm@deploy2002: urbanecm: Continuing with sync
* 08:23 urbanecm@deploy2002: urbanecm: Backport for [[gerrit:1114379[Growth] enwiki: Release Add Link to 10% of newcomers (T384551)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 08:21 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2020.codfw.wmnet with reason: host reimage
* 08:16 urbanecm@deploy2002: Started scap sync-world: Backport for [[gerrit:1114379[Growth] enwiki: Release Add Link to 10% of newcomers (T384551)]]
* 08:09 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T384592)', diff saved to https://phabricator.wikimedia.org/P72560 and previous config saved to /var/cache/conftool/dbconfig/20250128-080945-marostegui.json
* 08:07 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2155.codfw.wmnet with reason: Index rebuild
* 08:06 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2155.codfw.wmnet
* 08:06 slyngshede@dns1004: END - running authdns-update
* 08:04 slyngshede@dns1004: START - running authdns-update
* 08:00 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db[2186-2187].codfw.wmnet with reason: Index rebuild + upgrade
* 07:59 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2155.codfw.wmnet
* 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2026.codfw.wmnet
* 07:59 slyngshede@dns1004: END - running authdns-update
* 07:58 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2155 T382842', diff saved to https://phabricator.wikimedia.org/P72559 and previous config saved to /var/cache/conftool/dbconfig/20250128-075857-marostegui.json
* 07:57 slyngshede@dns1004: START - running authdns-update
* 07:56 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2026.codfw.wmnet
* 07:56 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1158 (T384592)', diff saved to https://phabricator.wikimedia.org/P72558 and previous config saved to /var/cache/conftool/dbconfig/20250128-075636-marostegui.json
* 07:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 07:56 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 07:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2020.codfw.wmnet with OS bookworm
* 07:51 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es1024.eqiad.wmnet
* 07:51 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:51 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es1024.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 07:50 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es1024.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 07:47 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2020.codfw.wmnet with reason: remove from cluster for reimage
* 07:35 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 07:29 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts es1024.eqiad.wmnet
* 07:27 marostegui@cumin1002: dbctl commit (dc=all): 'Remove es1024 from dbctl T384820', diff saved to https://phabricator.wikimedia.org/P72557 and previous config saved to /var/cache/conftool/dbconfig/20250128-072707-root.json
* 07:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 06:28 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1175 T384807', diff saved to https://phabricator.wikimedia.org/P72556 and previous config saved to /var/cache/conftool/dbconfig/20250128-062846-marostegui.json
* 06:25 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2205.codfw.wmnet with reason: Index rebuild
* 06:25 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1175.eqiad.wmnet with reason: Index rebuild
* 06:25 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2205.codfw.wmnet
* 06:25 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1175.eqiad.wmnet
* 06:19 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2205.codfw.wmnet
* 06:18 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1175.eqiad.wmnet
* 06:16 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2205.codfw.wmnet with reason: Index rebuild
* 06:12 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2205 T384807', diff saved to https://phabricator.wikimedia.org/P72555 and previous config saved to /var/cache/conftool/dbconfig/20250128-061230-marostegui.json
* 06:12 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1183.eqiad.wmnet with reason: Maintenance
* 05:11 aikochou@deploy2002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' .
* 05:06 mwpresync@deploy2002: Pruned MediaWiki: 1.44.0-wmf.11 (duration: 06m 25s)
* 05:03 mwpresync@deploy2002: Finished scap sync-world: testwikis to 1.44.0-wmf.14 refs T382365 (duration: 59m 38s)
* 04:58 aikochou@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' .
* 04:17 aikochou@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' .
* 04:03 mwpresync@deploy2002: Started scap sync-world: testwikis to 1.44.0-wmf.14 refs T382365
* 02:20 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 02:19 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
== 2025-01-27 ==
* 23:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2228 (T384592)', diff saved to https://phabricator.wikimedia.org/P72554 and previous config saved to /var/cache/conftool/dbconfig/20250127-231509-marostegui.json
* 23:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P72553 and previous config saved to /var/cache/conftool/dbconfig/20250127-230002-marostegui.json
* 22:54 ejegg: fundraising civicrm upgraded from to 
* 22:46 pt1979@cumin1002: START - Cookbook sre.hosts.dhcp for host cloudgw1003.eqiad.wmnet
* 22:46 pt1979@cumin1002: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cloudgw1003.eqiad.wmnet
* 22:45 pt1979@cumin1002: START - Cookbook sre.hosts.dhcp for host cloudgw1003.eqiad.wmnet
* 22:44 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P72552 and previous config saved to /var/cache/conftool/dbconfig/20250127-224455-marostegui.json
* 22:41 mstyles@deploy2002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 22:41 mstyles@deploy2002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 22:41 mstyles@deploy2002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 22:41 mstyles@deploy2002: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 22:40 mstyles@deploy2002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 22:40 mstyles@deploy2002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 22:39 mstyles@deploy2002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 22:39 mstyles@deploy2002: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 22:39 mstyles@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 22:38 mstyles@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 22:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2228 (T384592)', diff saved to https://phabricator.wikimedia.org/P72550 and previous config saved to /var/cache/conftool/dbconfig/20250127-222947-marostegui.json
* 21:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P72547 and previous config saved to /var/cache/conftool/dbconfig/20250127-215710-marostegui.json
* 21:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P72546 and previous config saved to /var/cache/conftool/dbconfig/20250127-214203-marostegui.json
* 21:33 cjming: end of UTC late backport window
* 21:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2223 (T384592)', diff saved to https://phabricator.wikimedia.org/P72544 and previous config saved to /var/cache/conftool/dbconfig/20250127-212656-marostegui.json
* 21:14 cjming@deploy2002: Finished scap sync-world: Backport for gerrit:1114396testwiki: Enable MetricsPlatform experiment enrollment overrides (T384728) (duration: 10m 30s)
* 21:08 cjming@deploy2002: phuedx, cjming: Continuing with sync
* 21:08 cjming@deploy2002: phuedx, cjming: Backport for gerrit:1114396testwiki: Enable MetricsPlatform experiment enrollment overrides (T384728) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:04 cjming@deploy2002: Started scap sync-world: Backport for gerrit:1114396testwiki: Enable MetricsPlatform experiment enrollment overrides (T384728)
* 21:04 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2223 (T384592)', diff saved to https://phabricator.wikimedia.org/P72543 and previous config saved to /var/cache/conftool/dbconfig/20250127-210415-marostegui.json
* 21:04 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2223.codfw.wmnet with reason: Maintenance
* 21:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2211 (T384592)', diff saved to https://phabricator.wikimedia.org/P72542 and previous config saved to /var/cache/conftool/dbconfig/20250127-210353-marostegui.json
* 21:01 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host logging-hd1005.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 20:50 vriley@cumin1002: START - Cookbook sre.hosts.provision for host logging-hd1005.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 20:48 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P72541 and previous config saved to /var/cache/conftool/dbconfig/20250127-204846-marostegui.json
* 20:45 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host logging-hd1005.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 20:45 vriley@cumin1002: START - Cookbook sre.hosts.provision for host logging-hd1005.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 20:36 papaul: power down logging-hd1005 for maintenance
* 20:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P72540 and previous config saved to /var/cache/conftool/dbconfig/20250127-203339-marostegui.json
* 19:59 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2211 (T384592)', diff saved to https://phabricator.wikimedia.org/P72538 and previous config saved to /var/cache/conftool/dbconfig/20250127-195939-marostegui.json
* 19:59 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2211.codfw.wmnet with reason: Maintenance
* 19:57 marostegui@cumin1002: dbctl commit (dc=all): 'db2194 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72537 and previous config saved to /var/cache/conftool/dbconfig/20250127-195736-root.json
* 19:42 marostegui@cumin1002: dbctl commit (dc=all): 'db2194 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72536 and previous config saved to /var/cache/conftool/dbconfig/20250127-194231-root.json
* 19:36 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2201.codfw.wmnet with reason: Maintenance
* 19:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2192 (T384592)', diff saved to https://phabricator.wikimedia.org/P72535 and previous config saved to /var/cache/conftool/dbconfig/20250127-193625-marostegui.json
* 19:27 marostegui@cumin1002: dbctl commit (dc=all): 'db2194 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72534 and previous config saved to /var/cache/conftool/dbconfig/20250127-192725-root.json
* 19:25 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 19:25 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 19:24 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 19:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P72533 and previous config saved to /var/cache/conftool/dbconfig/20250127-192118-marostegui.json
* 19:19 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 19:18 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 19:18 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 19:17 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 19:17 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding puppetserver2004 to codfw - jhancock@cumin2002"
* 19:17 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding puppetserver2004 to codfw - jhancock@cumin2002"
* 19:13 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 19:13 ejegg: deployed fallback SMTP settings for Fundraising CiviCRM
* 19:12 marostegui@cumin1002: dbctl commit (dc=all): 'db2194 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72532 and previous config saved to /var/cache/conftool/dbconfig/20250127-191220-root.json
* 19:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P72531 and previous config saved to /var/cache/conftool/dbconfig/20250127-190611-marostegui.json
* 18:59 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:59 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt pay-lb1002~ - vriley@cumin1002"
* 18:59 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt pay-lb1002~ - vriley@cumin1002"
* 18:57 marostegui@cumin1002: dbctl commit (dc=all): 'db2194 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72530 and previous config saved to /var/cache/conftool/dbconfig/20250127-185715-root.json
* 18:56 vriley@cumin1002: START - Cookbook sre.dns.netbox
* 18:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2192 (T384592)', diff saved to https://phabricator.wikimedia.org/P72529 and previous config saved to /var/cache/conftool/dbconfig/20250127-185104-marostegui.json
* 18:48 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2194', diff saved to https://phabricator.wikimedia.org/P72528 and previous config saved to /var/cache/conftool/dbconfig/20250127-184839-marostegui.json
* 18:46 marostegui@cumin1002: dbctl commit (dc=all): 'db2194 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P72527 and previous config saved to /var/cache/conftool/dbconfig/20250127-184642-root.json
* 18:35 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:35 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt pay-lb1001 - vriley@cumin1002"
* 18:35 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt pay-lb1001 - vriley@cumin1002"
* 18:35 marostegui@cumin1002: dbctl commit (dc=all): 'db1157 (re)pooling @ 100%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72526 and previous config saved to /var/cache/conftool/dbconfig/20250127-183519-root.json
* 18:34 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2192 (T384592)', diff saved to https://phabricator.wikimedia.org/P72525 and previous config saved to /var/cache/conftool/dbconfig/20250127-183417-marostegui.json
* 18:34 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2192.codfw.wmnet with reason: Maintenance
* 18:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T384592)', diff saved to https://phabricator.wikimedia.org/P72524 and previous config saved to /var/cache/conftool/dbconfig/20250127-183355-marostegui.json
* 18:31 vriley@cumin1002: START - Cookbook sre.dns.netbox
* 18:27 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudgw1004.eqiad.wmnet with OS bookworm
* 18:20 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudgw1004.eqiad.wmnet with OS bookworm
* 18:20 marostegui@cumin1002: dbctl commit (dc=all): 'db1157 (re)pooling @ 75%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72523 and previous config saved to /var/cache/conftool/dbconfig/20250127-182014-root.json
* 18:18 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P72522 and previous config saved to /var/cache/conftool/dbconfig/20250127-181847-marostegui.json
* 18:16 reedy@deploy2002: Synchronized php-1.44.0-wmf.13/extensions/CommonsMetadata/: T384853 T384854 (duration: 10m 45s)
* 18:10 tchin@deploy2002: Finished deploy [airflow-dags/analytics@c49f40b]: Deploying airflow for T357684 (duration: 01m 01s)
* 18:10 tchin@deploy2002: Started deploy [airflow-dags/analytics@c49f40b]: Deploying airflow for T357684
* 18:05 marostegui@cumin1002: dbctl commit (dc=all): 'db1157 (re)pooling @ 50%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72521 and previous config saved to /var/cache/conftool/dbconfig/20250127-180509-root.json
* 18:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P72520 and previous config saved to /var/cache/conftool/dbconfig/20250127-180341-marostegui.json
* 17:55 andrew@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudgw1003.eqiad.wmnet with OS bookworm
* 17:50 marostegui@cumin1002: dbctl commit (dc=all): 'db1157 (re)pooling @ 25%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72518 and previous config saved to /var/cache/conftool/dbconfig/20250127-175004-root.json
* 17:48 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T384592)', diff saved to https://phabricator.wikimedia.org/P72517 and previous config saved to /var/cache/conftool/dbconfig/20250127-174833-marostegui.json
* 17:48 jgiannelos@deploy2002: helmfile [staging] DONE helmfile.d/services/kartotherian: sync
* 17:48 jgiannelos@deploy2002: helmfile [staging] START helmfile.d/services/kartotherian: sync
* 17:34 marostegui@cumin1002: dbctl commit (dc=all): 'db1157 (re)pooling @ 10%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72516 and previous config saved to /var/cache/conftool/dbconfig/20250127-173458-root.json
* 17:30 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:28 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 17:28 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2178 (T384592)', diff saved to https://phabricator.wikimedia.org/P72515 and previous config saved to /var/cache/conftool/dbconfig/20250127-172814-marostegui.json
* 17:28 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2178.codfw.wmnet with reason: Maintenance
* 17:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2171 (T384592)', diff saved to https://phabricator.wikimedia.org/P72514 and previous config saved to /var/cache/conftool/dbconfig/20250127-172752-marostegui.json
* 17:19 marostegui@cumin1002: dbctl commit (dc=all): 'db1157 (re)pooling @ 5%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72513 and previous config saved to /var/cache/conftool/dbconfig/20250127-171953-root.json
* 17:18 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:18 swfrench@deploy2002: Finished scap sync-world: Backport for gerrit:1113567Enroll 1% of client sessions in PHP 8.1 (T383845) (duration: 14m 02s)
* 17:15 pt1979@cumin2002: START - Cookbook sre.dns.netbox
* 17:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P72512 and previous config saved to /var/cache/conftool/dbconfig/20250127-171245-marostegui.json
* 17:10 swfrench@deploy2002: swfrench: Continuing with sync
* 17:09 tchin@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
* 17:09 tchin@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
* 17:09 swfrench@deploy2002: swfrench: Backport for gerrit:1113567Enroll 1% of client sessions in PHP 8.1 (T383845) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 17:04 marostegui@cumin1002: dbctl commit (dc=all): 'db1157 (re)pooling @ 4%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72511 and previous config saved to /var/cache/conftool/dbconfig/20250127-170448-root.json
* 17:04 swfrench@deploy2002: Started scap sync-world: Backport for gerrit:1113567Enroll 1% of client sessions in PHP 8.1 (T383845)
* 16:58 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1159-1165].eqiad.wmnet
* 16:58 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1159-1165].eqiad.wmnet
* 16:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P72510 and previous config saved to /var/cache/conftool/dbconfig/20250127-165738-marostegui.json
* 16:56 klausman@deploy2002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 16:54 klausman@deploy2002: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 16:49 marostegui@cumin1002: dbctl commit (dc=all): 'db2227 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72509 and previous config saved to /var/cache/conftool/dbconfig/20250127-164955-root.json
* 16:49 marostegui@cumin1002: dbctl commit (dc=all): 'db1157 (re)pooling @ 3%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72508 and previous config saved to /var/cache/conftool/dbconfig/20250127-164942-root.json
* 16:48 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1164.eqiad.wmnet with OS bookworm
* 16:47 jhancock@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['ms-be2075']
* 16:46 jhancock@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be2075']
* 16:45 elukey@deploy2002: helmfile [staging] DONE helmfile.d/services/kartotherian: sync
* 16:44 elukey@deploy2002: helmfile [staging] START helmfile.d/services/kartotherian: sync
* 16:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2171 (T384592)', diff saved to https://phabricator.wikimedia.org/P72507 and previous config saved to /var/cache/conftool/dbconfig/20250127-164231-marostegui.json
* 16:35 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudgw1003.eqiad.wmnet with OS bookworm
* 16:34 marostegui@cumin1002: dbctl commit (dc=all): 'db2227 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72506 and previous config saved to /var/cache/conftool/dbconfig/20250127-163449-root.json
* 16:34 marostegui@cumin1002: dbctl commit (dc=all): 'db1157 (re)pooling @ 2%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72505 and previous config saved to /var/cache/conftool/dbconfig/20250127-163437-root.json
* 16:32 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl[1002-1003].eqiad.wmnet
* 16:32 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-ctrl[1002-1003].eqiad.wmnet
* 16:32 jayme@cumin1002: START - Cookbook sre.hosts.remove-downtime for wikikube-ctrl[1002-1003].eqiad.wmnet
* 16:32 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl[1002-1003].eqiad.wmnet
* 16:32 klausman@deploy2002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 16:32 klausman@deploy2002: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 16:28 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1164.eqiad.wmnet with reason: host reimage
* 16:25 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1164.eqiad.wmnet with reason: host reimage
* 16:20 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2171 (T384592)', diff saved to https://phabricator.wikimedia.org/P72504 and previous config saved to /var/cache/conftool/dbconfig/20250127-162018-marostegui.json
* 16:20 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2171.codfw.wmnet with reason: Maintenance
* 16:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T384592)', diff saved to https://phabricator.wikimedia.org/P72503 and previous config saved to /var/cache/conftool/dbconfig/20250127-161956-marostegui.json
* 16:19 marostegui@cumin1002: dbctl commit (dc=all): 'db2227 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72502 and previous config saved to /var/cache/conftool/dbconfig/20250127-161944-root.json
* 16:19 marostegui@cumin1002: dbctl commit (dc=all): 'db1157 (re)pooling @ 1%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72501 and previous config saved to /var/cache/conftool/dbconfig/20250127-161932-root.json
* 16:18 swfrench@deploy2002: Finished scap sync-world: Backport for gerrit:1113566Enroll 0.1% of client sessions in PHP 8.1 (T383845) (duration: 18m 11s)
* 16:16 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2025.codfw.wmnet to cluster codfw and group D
* 16:13 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2025.codfw.wmnet to cluster codfw and group D
* 16:12 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti2020
* 16:12 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ganeti2020
* 16:11 swfrench@deploy2002: swfrench: Continuing with sync
* 16:09 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1164
* 16:09 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1164
* 16:09 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1164.eqiad.wmnet with OS bookworm
* 16:08 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1163.eqiad.wmnet with OS bookworm
* 16:08 kamila@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1164.eqiad.wmnet with OS bookworm
* 16:07 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2194.codfw.wmnet with reason: Index rebuild T384807
* 16:07 cstone: SmashPig upgraded from to 
* 16:05 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1165.eqiad.wmnet with OS bookworm
* 16:05 swfrench@deploy2002: swfrench: Backport for gerrit:1113566Enroll 0.1% of client sessions in PHP 8.1 (T383845) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 16:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P72499 and previous config saved to /var/cache/conftool/dbconfig/20250127-160449-marostegui.json
* 16:04 marostegui@cumin1002: dbctl commit (dc=all): 'db2227 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72498 and previous config saved to /var/cache/conftool/dbconfig/20250127-160438-root.json
* 16:03 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2194', diff saved to https://phabricator.wikimedia.org/P72497 and previous config saved to /var/cache/conftool/dbconfig/20250127-160300-marostegui.json
* 16:02 marostegui@cumin1002: dbctl commit (dc=all): 'db2194 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P72496 and previous config saved to /var/cache/conftool/dbconfig/20250127-160237-root.json
* 16:02 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2194.codfw.wmnet with reason: Index rebuild
* 16:01 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1159.eqiad.wmnet with OS bookworm
* 16:01 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2194.codfw.wmnet
* 16:00 swfrench@deploy2002: Started scap sync-world: Backport for gerrit:1113566Enroll 0.1% of client sessions in PHP 8.1 (T383845)
* 15:58 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1162.eqiad.wmnet with OS bookworm
* 15:56 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2194.codfw.wmnet
* 15:56 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2194 T384807', diff saved to https://phabricator.wikimedia.org/P72495 and previous config saved to /var/cache/conftool/dbconfig/20250127-155613-marostegui.json
* 15:54 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1161.eqiad.wmnet with OS bookworm
* 15:53 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1160.eqiad.wmnet with OS bookworm
* 15:52 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:52 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt frnetmon1002 - vriley@cumin1002"
* 15:52 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt frnetmon1002 - vriley@cumin1002"
* 15:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P72494 and previous config saved to /var/cache/conftool/dbconfig/20250127-154942-marostegui.json
* 15:49 marostegui@cumin1002: dbctl commit (dc=all): 'db2227 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72493 and previous config saved to /var/cache/conftool/dbconfig/20250127-154933-root.json
* 15:49 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1163.eqiad.wmnet with reason: host reimage
* 15:48 vriley@cumin1002: START - Cookbook sre.dns.netbox
* 15:46 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1165.eqiad.wmnet with reason: host reimage
* 15:42 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1159.eqiad.wmnet with reason: host reimage
* 15:39 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1162.eqiad.wmnet with reason: host reimage
* 15:36 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1161.eqiad.wmnet with reason: host reimage
* 15:35 taavi@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cloudlb2002-dev.private.codfw.wikimedia.cloud on all recursors
* 15:35 taavi@cumin1002: START - Cookbook sre.dns.wipe-cache cloudlb2002-dev.private.codfw.wikimedia.cloud on all recursors
* 15:35 taavi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:35 taavi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add v6 cloud-private address for cloudlb2002-dev - taavi@cumin1002"
* 15:35 taavi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add v6 cloud-private address for cloudlb2002-dev - taavi@cumin1002"
* 15:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T384592)', diff saved to https://phabricator.wikimedia.org/P72491 and previous config saved to /var/cache/conftool/dbconfig/20250127-153435-marostegui.json
* 15:32 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1160.eqiad.wmnet with reason: host reimage
* 15:30 taavi@cumin1002: START - Cookbook sre.dns.netbox
* 15:27 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1159.eqiad.wmnet with reason: host reimage
* 15:26 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1163.eqiad.wmnet with reason: host reimage
* 15:26 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1165.eqiad.wmnet with reason: host reimage
* 15:26 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1162.eqiad.wmnet with reason: host reimage
* 15:26 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1161.eqiad.wmnet with reason: host reimage
* 15:26 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1160.eqiad.wmnet with reason: host reimage
* 15:18 taavi@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cloudlb2003-dev.private.codfw.wikimedia.cloud on all recursors
* 15:18 taavi@cumin1002: START - Cookbook sre.dns.wipe-cache cloudlb2003-dev.private.codfw.wikimedia.cloud on all recursors
* 15:16 jelto@cumin1002: END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on A:gitlab-runner
* 15:15 taavi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:15 taavi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add v6 cloud-private address for cloudlb2003-dev - taavi@cumin1002"
* 15:15 taavi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add v6 cloud-private address for cloudlb2003-dev - taavi@cumin1002"
* 15:11 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1159
* 15:11 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1159
* 15:11 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1159.eqiad.wmnet with OS bookworm
* 15:11 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1165
* 15:11 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1165
* 15:11 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1165.eqiad.wmnet with OS bookworm
* 15:11 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1164
* 15:11 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1164
* 15:11 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1164.eqiad.wmnet with OS bookworm
* 15:11 taavi@cumin1002: START - Cookbook sre.dns.netbox
* 15:11 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1163
* 15:11 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1163
* 15:10 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1163.eqiad.wmnet with OS bookworm
* 15:10 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1162
* 15:10 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1162
* 15:10 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1162.eqiad.wmnet with OS bookworm
* 15:10 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1161
* 15:10 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1161
* 15:10 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1161.eqiad.wmnet with OS bookworm
* 15:10 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1160
* 15:10 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1160
* 15:10 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1160.eqiad.wmnet with OS bookworm
* 15:10 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2157 (T384592)', diff saved to https://phabricator.wikimedia.org/P72490 and previous config saved to /var/cache/conftool/dbconfig/20250127-151007-marostegui.json
* 15:10 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2157.codfw.wmnet with reason: Maintenance
* 15:09 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl[1002-1003].eqiad.wmnet
* 15:09 jayme@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl[1002-1003].eqiad.wmnet with reason: Depooled via sre.k8s.pool-depool-node
* 15:09 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl[1002-1003].eqiad.wmnet
* 15:07 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1159.eqiad.wmnet wikikube-worker1160.eqiad.wmnet wikikube-worker1161.eqiad.wmnet wikikube-worker1162.eqiad.wmnet wikikube-worker1163.eqiad.wmnet wikikube-worker1164.eqiad.wmnet wikikube-worker1165.eqiad.wmnet on all recursors
* 15:07 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1159.eqiad.wmnet wikikube-worker1160.eqiad.wmnet wikikube-worker1161.eqiad.wmnet wikikube-worker1162.eqiad.wmnet wikikube-worker1163.eqiad.wmnet wikikube-worker1164.eqiad.wmnet wikikube-worker1165.eqiad.wmnet on all recursors
* 15:07 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1024 to wikikube-worker1165
* 15:06 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1165
* 15:04 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1165
* 15:04 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:04 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1024 to wikikube-worker1165 - kamila@cumin1002"
* 15:04 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1024 to wikikube-worker1165 - kamila@cumin1002"
* 15:01 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1023 to wikikube-worker1164
* 15:01 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1164
* 15:00 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 15:00 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1164
* 14:59 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:59 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1023 to wikikube-worker1164 - kamila@cumin1002"
* 14:59 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1024 to wikikube-worker1165
* 14:59 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1023 to wikikube-worker1164 - kamila@cumin1002"
* 14:58 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1022 to wikikube-worker1163
* 14:57 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1163
* 14:57 zabe@deploy2002: sync-world aborted: T384614 T183490 (duration: 17m 07s)
* 14:56 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 14:56 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1163
* 14:55 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:55 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1022 to wikikube-worker1163 - kamila@cumin1002"
* 14:55 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1023 to wikikube-worker1164
* 14:55 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1022 to wikikube-worker1163 - kamila@cumin1002"
* 14:53 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1021 to wikikube-worker1162
* 14:52 tchin@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 14:52 tchin@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 14:52 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1162
* 14:51 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 14:51 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1162
* 14:51 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:51 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1021 to wikikube-worker1162 - kamila@cumin1002"
* 14:51 tchin@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 14:51 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1022 to wikikube-worker1163
* 14:51 tchin@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 14:51 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1021 to wikikube-worker1162 - kamila@cumin1002"
* 14:50 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
* 14:50 tchin@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 14:50 tchin@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 14:49 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1020 to wikikube-worker1161
* 14:48 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1161
* 14:47 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 14:47 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1161
* 14:47 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:47 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1020 to wikikube-worker1161 - kamila@cumin1002"
* 14:47 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1021 to wikikube-worker1162
* 14:47 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1020 to wikikube-worker1161 - kamila@cumin1002"
* 14:45 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1019 to wikikube-worker1160
* 14:45 zabe@deploy2002: zabe: Continuing with sync
* 14:45 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2025.codfw.wmnet
* 14:45 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1160
* 14:44 zabe@deploy2002: zabe: T384614 T183490 synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:43 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 14:43 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1160
* 14:43 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:43 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1019 to wikikube-worker1160 - kamila@cumin1002"
* 14:43 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1020 to wikikube-worker1161
* 14:43 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1019 to wikikube-worker1160 - kamila@cumin1002"
* 14:40 zabe@deploy2002: Started scap sync-world: T384614 T183490
* 14:39 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1018 to wikikube-worker1159
* 14:38 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1159
* 14:38 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 14:37 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1159
* 14:37 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:37 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1018 to wikikube-worker1159 - kamila@cumin1002"
* 14:37 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1019 to wikikube-worker1160
* 14:37 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1018 to wikikube-worker1159 - kamila@cumin1002"
* 14:37 jelto@cumin1002: START - Cookbook sre.gitlab.reboot-runner rolling reboot on A:gitlab-runner
* 14:36 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2025.codfw.wmnet
* 14:33 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 14:33 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1018 to wikikube-worker1159
* 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2025.codfw.wmnet with OS bookworm
* 14:32 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1245.eqiad.wmnet with reason: Maintenance
* 14:32 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host parse[1018-1024].eqiad.wmnet
* 14:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230 (T384592)', diff saved to https://phabricator.wikimedia.org/P72489 and previous config saved to /var/cache/conftool/dbconfig/20250127-143211-marostegui.json
* 14:31 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 14:28 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host parse[1018-1024].eqiad.wmnet
* 14:22 zabe@deploy2002: tsev, zabe, anzx: Continuing with sync
* 14:21 zabe@deploy2002: tsev, zabe, anzx: Backport for gerrit:1113996Add ios.article_link_interaction stream to config (T382031), gerrit:1114378srwiki: add incubator as importsource (T384069), gerrit:1114372enwiki: temporary lift of IP cap for 31 January and 1 February 2025 (T384680) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P72488 and previous config saved to /var/cache/conftool/dbconfig/20250127-141704-marostegui.json
* 14:16 marostegui@cumin1002: dbctl commit (dc=all): 'db2182 (re)pooling @ 100%: Repooling T384801', diff saved to https://phabricator.wikimedia.org/P72487 and previous config saved to /var/cache/conftool/dbconfig/20250127-141616-root.json
* 14:13 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1013.eqiad.wmnet with reason: host reimage
* 14:10 andrew@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1013.eqiad.wmnet with reason: host reimage
* 14:10 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2025.codfw.wmnet with reason: host reimage
* 14:09 zabe@deploy2002: Started scap sync-world: Backport for gerrit:1113996Add ios.article_link_interaction stream to config (T382031), gerrit:1114378srwiki: add incubator as importsource (T384069), gerrit:1114372enwiki: temporary lift of IP cap for 31 January and 1 February 2025 (T384680)
* 14:06 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2025.codfw.wmnet with reason: host reimage
* 14:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P72486 and previous config saved to /var/cache/conftool/dbconfig/20250127-140157-marostegui.json
* 14:01 marostegui@cumin1002: dbctl commit (dc=all): 'db2182 (re)pooling @ 75%: Repooling T384801', diff saved to https://phabricator.wikimedia.org/P72485 and previous config saved to /var/cache/conftool/dbconfig/20250127-140111-root.json
* 13:53 gmodena@deploy2002: Finished deploy [airflow-dags/search@3c004c1]: syncing artifacts (duration: 01m 04s)
* 13:53 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 13:53 tchin@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 13:53 tchin@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 13:53 gmodena@deploy2002: Started deploy [airflow-dags/search@3c004c1]: syncing artifacts
* 13:53 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 13:46 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230 (T384592)', diff saved to https://phabricator.wikimedia.org/P72484 and previous config saved to /var/cache/conftool/dbconfig/20250127-134650-marostegui.json
* 13:46 marostegui@cumin1002: dbctl commit (dc=all): 'db2182 (re)pooling @ 50%: Repooling T384801', diff saved to https://phabricator.wikimedia.org/P72483 and previous config saved to /var/cache/conftool/dbconfig/20250127-134605-root.json
* 13:35 federico3: Removing db2140 from zarcillo T384480
* 13:34 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1157.eqiad.wmnet with reason: Index rebuild
* 13:34 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1157.eqiad.wmnet
* 13:33 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1013.eqiad.wmnet with reason: host reimage
* 13:32 moritzm: installing runc security updates on bullseye
* 13:31 marostegui@cumin1002: dbctl commit (dc=all): 'db2182 (re)pooling @ 25%: Repooling T384801', diff saved to https://phabricator.wikimedia.org/P72482 and previous config saved to /var/cache/conftool/dbconfig/20250127-133059-root.json
* 13:28 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1230 (T384592)', diff saved to https://phabricator.wikimedia.org/P72481 and previous config saved to /var/cache/conftool/dbconfig/20250127-132806-marostegui.json
* 13:28 fceratto@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2140.codfw.wmnet
* 13:28 fceratto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:28 fceratto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2140.codfw.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1002"
* 13:28 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1230.eqiad.wmnet with reason: Maintenance
* 13:27 fceratto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2140.codfw.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1002"
* 13:27 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1157.eqiad.wmnet
* 13:27 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1157 T384807', diff saved to https://phabricator.wikimedia.org/P72480 and previous config saved to /var/cache/conftool/dbconfig/20250127-132710-marostegui.json
* 13:26 andrew@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1013.eqiad.wmnet with reason: host reimage
* 13:25 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2025.codfw.wmnet with OS bookworm
* 13:23 fceratto@cumin1002: START - Cookbook sre.dns.netbox
* 13:18 fceratto@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2140.codfw.wmnet
* 13:15 marostegui@cumin1002: dbctl commit (dc=all): 'db2182 (re)pooling @ 10%: Repooling T384801', diff saved to https://phabricator.wikimedia.org/P72479 and previous config saved to /var/cache/conftool/dbconfig/20250127-131554-root.json
* 13:13 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2025.codfw.wmnet with reason: remove from cluster for reimage
* 13:11 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 13:10 andrew@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 13:08 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1216.eqiad.wmnet with reason: Maintenance
* 13:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210 (T384592)', diff saved to https://phabricator.wikimedia.org/P72478 and previous config saved to /var/cache/conftool/dbconfig/20250127-130841-marostegui.json
* 13:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2025.codfw.wmnet
* 12:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P72477 and previous config saved to /var/cache/conftool/dbconfig/20250127-125334-marostegui.json
* 12:53 marostegui@cumin1002: dbctl commit (dc=all): 'db1223 (re)pooling @ 100%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72476 and previous config saved to /var/cache/conftool/dbconfig/20250127-125301-root.json
* 12:39 root@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1171.eqiad.wmnet with OS bookworm
* 12:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P72474 and previous config saved to /var/cache/conftool/dbconfig/20250127-123827-marostegui.json
* 12:37 marostegui@cumin1002: dbctl commit (dc=all): 'db1223 (re)pooling @ 75%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72473 and previous config saved to /var/cache/conftool/dbconfig/20250127-123754-root.json
* 12:29 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2227.codfw.wmnet with reason: Index rebuild
* 12:25 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2227.codfw.wmnet
* 12:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210 (T384592)', diff saved to https://phabricator.wikimedia.org/P72472 and previous config saved to /var/cache/conftool/dbconfig/20250127-122320-marostegui.json
* 12:22 marostegui@cumin1002: dbctl commit (dc=all): 'db1223 (re)pooling @ 50%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72471 and previous config saved to /var/cache/conftool/dbconfig/20250127-122249-root.json
* 12:19 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2227.codfw.wmnet
* 12:18 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2227 T384807', diff saved to https://phabricator.wikimedia.org/P72470 and previous config saved to /var/cache/conftool/dbconfig/20250127-121843-marostegui.json
* 12:16 root@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1171.eqiad.wmnet with reason: host reimage
* 12:12 root@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1171.eqiad.wmnet with reason: host reimage
* 12:08 moritzm: installing git-lfs security updates
* 12:07 marostegui@cumin1002: dbctl commit (dc=all): 'db1223 (re)pooling @ 25%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72469 and previous config saved to /var/cache/conftool/dbconfig/20250127-120744-root.json
* 12:05 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1210 (T384592)', diff saved to https://phabricator.wikimedia.org/P72468 and previous config saved to /var/cache/conftool/dbconfig/20250127-120507-marostegui.json
* 12:05 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1210.eqiad.wmnet with reason: Maintenance
* 12:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T384592)', diff saved to https://phabricator.wikimedia.org/P72467 and previous config saved to /var/cache/conftool/dbconfig/20250127-120445-marostegui.json
* 11:58 root@cumin1002: START - Cookbook sre.hosts.reimage for host db1171.eqiad.wmnet with OS bookworm
* 11:56 root@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host db1171.eqiad.wmnet with OS bookworm
* 11:52 marostegui@cumin1002: dbctl commit (dc=all): 'db1223 (re)pooling @ 10%: Repooling T384807', diff saved to https://phabricator.wikimedia.org/P72466 and previous config saved to /var/cache/conftool/dbconfig/20250127-115239-root.json
* 11:50 jiji@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P72465 and previous config saved to /var/cache/conftool/dbconfig/20250127-114938-marostegui.json
* 11:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rpki1001.eqiad.wmnet with OS bookworm
* 11:45 jiji@deploy2002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:45 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1215.eqiad.wmnet
* 11:44 jiji@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:41 jiji@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:41 jiji@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
* 11:40 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2025.codfw.wmnet
* 11:40 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster2003.codfw.wmnet to plain
* 11:39 marostegui: Upgrade and reboot zarcillo/orchestrator database db1215
* 11:39 jiji@deploy2002: helmfile [codfw] START helmfile.d/services/mw-web: apply
* 11:39 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1215.eqiad.wmnet
* 11:39 jiji@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
* 11:38 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster2003.codfw.wmnet to plain
* 11:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2025.codfw.wmnet
* 11:36 jiji@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 11:35 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2025.codfw.wmnet
* 11:35 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster2003.codfw.wmnet to drbd
* 11:34 topranks: rebooting cr1-magru to complete upgrade T384774
* 11:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P72463 and previous config saved to /var/cache/conftool/dbconfig/20250127-113431-marostegui.json
* 11:33 tchin@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 11:33 tchin@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 11:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 11:27 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 11:25 aikochou@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' .
* 11:20 topranks: installing updated JunOS image on cr1-magru T384774
* 11:19 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster2003.codfw.wmnet to drbd
* 11:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T384592)', diff saved to https://phabricator.wikimedia.org/P72462 and previous config saved to /var/cache/conftool/dbconfig/20250127-111924-marostegui.json
* 11:17 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host rpki1001.eqiad.wmnet with OS bookworm
* 11:14 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2025.codfw.wmnet
* 11:14 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2025.codfw.wmnet
* 11:04 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2185.codfw.wmnet
* 11:04 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host rpki1001.eqiad.wmnet with OS bookworm
* 10:59 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2185.codfw.wmnet
* 10:59 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1200 (T384592)', diff saved to https://phabricator.wikimedia.org/P72461 and previous config saved to /var/cache/conftool/dbconfig/20250127-105944-marostegui.json
* 10:59 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1200.eqiad.wmnet with reason: Maintenance
* 10:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T384592)', diff saved to https://phabricator.wikimedia.org/P72460 and previous config saved to /var/cache/conftool/dbconfig/20250127-105922-marostegui.json
* 10:56 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-b[3-4]-magru.mgmt with reason: upgrading JunOS on magru core routers
* 10:47 topranks: rebooting cr2-magru to complete upgrade T384774
* 10:44 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P72459 and previous config saved to /var/cache/conftool/dbconfig/20250127-104415-marostegui.json
* 10:43 vgutierrez: testing pybal 1.15.15 in lvs4010
* 10:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2025.codfw.wmnet
* 10:43 root@cumin1002: START - Cookbook sre.hosts.reimage for host db1171.eqiad.wmnet with OS bookworm
* 10:42 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2025.codfw.wmnet
* 10:42 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test1004.wikimedia.org
* 10:38 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host idp-test1004.wikimedia.org
* 10:37 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1171.eqiad.wmnet with reason: reimage
* 10:36 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host rpki1001.eqiad.wmnet with OS bookworm
* 10:36 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host rpki1001.eqiad.wmnet with OS bookworm
* 10:34 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host rpki1001.eqiad.wmnet with OS bookworm
* 10:34 fabfur: installing haproxykafka on esams (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1114329) (T378578)
* 10:33 jmm@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host rpki1001.eqiad.wmnet with OS bookworm
* 10:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P72458 and previous config saved to /var/cache/conftool/dbconfig/20250127-102908-marostegui.json
* 10:26 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1024 T384820', diff saved to https://phabricator.wikimedia.org/P72457 and previous config saved to /var/cache/conftool/dbconfig/20250127-102657-marostegui.json
* 10:24 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test2004.wikimedia.org
* 10:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host idp-test2004.wikimedia.org
* 10:20 topranks: installing updated JunOS image on cr2-magru T384774
* 10:16 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore[1007,1009].eqiad.wmnet with reason: Index rebuild + upgrade
* 10:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T384592)', diff saved to https://phabricator.wikimedia.org/P72456 and previous config saved to /var/cache/conftool/dbconfig/20250127-101401-marostegui.json
* 10:13 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test2005.wikimedia.org
* 10:11 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore[1007,1009].eqiad.wmnet with reason: Index rebuild + upgrade
* 10:09 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host idp-test2005.wikimedia.org
* 10:07 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2020.codfw.wmnet
* 10:04 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on cr[1-2]-magru,cr[1-2]-magru IPv6 with reason: upgrading JunOS on magru core routers
* 09:54 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1185 (T384592)', diff saved to https://phabricator.wikimedia.org/P72455 and previous config saved to /var/cache/conftool/dbconfig/20250127-095416-marostegui.json
* 09:54 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1185.eqiad.wmnet with reason: Maintenance
* 09:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T384592)', diff saved to https://phabricator.wikimedia.org/P72454 and previous config saved to /var/cache/conftool/dbconfig/20250127-095354-marostegui.json
* 09:47 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host rpki1001.eqiad.wmnet with OS bookworm
* 09:47 moritzm: reimaging rpki1001 to bookworm
* 09:42 marostegui@cumin1002: dbctl commit (dc=all): 'db1241 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72453 and previous config saved to /var/cache/conftool/dbconfig/20250127-094206-root.json
* 09:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P72452 and previous config saved to /var/cache/conftool/dbconfig/20250127-093847-marostegui.json
* 09:32 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-presto1001.eqiad.wmnet
* 09:27 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host an-test-presto1001.eqiad.wmnet
* 09:27 marostegui@cumin1002: dbctl commit (dc=all): 'db1241 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72451 and previous config saved to /var/cache/conftool/dbconfig/20250127-092701-root.json
* 09:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P72450 and previous config saved to /var/cache/conftool/dbconfig/20250127-092340-marostegui.json
* 09:11 marostegui@cumin1002: dbctl commit (dc=all): 'db1241 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72449 and previous config saved to /var/cache/conftool/dbconfig/20250127-091155-root.json
* 09:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T384592)', diff saved to https://phabricator.wikimedia.org/P72448 and previous config saved to /var/cache/conftool/dbconfig/20250127-090833-marostegui.json
* 09:02 jmm@dns1004: END - running authdns-update
* 09:00 jmm@dns1004: START - running authdns-update
* 08:56 moritzm: installing net-tools bugfix updates on bullseye
* 08:56 marostegui@cumin1002: dbctl commit (dc=all): 'db1241 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72447 and previous config saved to /var/cache/conftool/dbconfig/20250127-085650-root.json
* 08:56 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1223.eqiad.wmnet with reason: Index rebuild
* 08:54 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1223.eqiad.wmnet
* 08:50 root@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1223.eqiad.wmnet with reason: Index rebuild + upgrade
* 08:49 marostegui: Upgrade db1223 T384807
* 08:49 root@cumin1002: START - Cookbook sre.mysql.upgrade for db1223.eqiad.wmnet
* 08:48 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1223', diff saved to https://phabricator.wikimedia.org/P72446 and previous config saved to /var/cache/conftool/dbconfig/20250127-084857-marostegui.json
* 08:47 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1161 (T384592)', diff saved to https://phabricator.wikimedia.org/P72445 and previous config saved to /var/cache/conftool/dbconfig/20250127-084713-marostegui.json
* 08:47 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 08:46 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 08:46 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1159 (T384592)', diff saved to https://phabricator.wikimedia.org/P72444 and previous config saved to /var/cache/conftool/dbconfig/20250127-084633-marostegui.json
* 08:42 moritzm: installing gtk+3.0 security updates
* 08:42 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es1023.eqiad.wmnet
* 08:42 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:42 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es1023.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 08:42 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es1023.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 08:41 marostegui@cumin1002: dbctl commit (dc=all): 'db1241 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72443 and previous config saved to /var/cache/conftool/dbconfig/20250127-084145-root.json
* 08:31 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P72442 and previous config saved to /var/cache/conftool/dbconfig/20250127-083126-marostegui.json
* 08:30 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 08:24 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts es1023.eqiad.wmnet
* 08:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P72441 and previous config saved to /var/cache/conftool/dbconfig/20250127-081619-marostegui.json
* 08:12 moritzm: installing rsync regression updates on bullseye
* 08:02 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2020.codfw.wmnet
* 08:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1159 (T384592)', diff saved to https://phabricator.wikimedia.org/P72440 and previous config saved to /var/cache/conftool/dbconfig/20250127-080112-marostegui.json
* 07:46 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp2004.wikimedia.org
* 07:44 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host idp2004.wikimedia.org
* 07:40 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1159 (T384592)', diff saved to https://phabricator.wikimedia.org/P72439 and previous config saved to /var/cache/conftool/dbconfig/20250127-074030-marostegui.json
* 07:40 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1159.eqiad.wmnet with reason: Maintenance
* 07:35 moritzm: installing tomcat10 security updates
* 07:25 slyngshede@dns1004: END - running authdns-update
* 07:23 slyngshede@dns1004: START - running authdns-update
* 06:54 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2204.codfw.wmnet with reason: Maintenance
* 06:24 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1162.eqiad.wmnet with reason: Maintenance
* 06:23 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1241.eqiad.wmnet with reason: Index rebuild + upgrade
== 2025-01-26 ==
* 21:40 mvernon@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2182.codfw.wmnet with reason: needs index repair
* 21:39 mvernon@cumin2002: dbctl commit (dc=all): 'Depool db2182', diff saved to https://phabricator.wikimedia.org/P72438 and previous config saved to /var/cache/conftool/dbconfig/20250126-213912-mvernon.json
* 20:30 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1241.eqiad.wmnet with reason: Index rebuild + upgrade
* 17:43 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 17:43 andrew@cumin1002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 17:43 andrew@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 17:37 andrew@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 17:18 andrew@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 17:18 andrew@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 17:18 andrew@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 17:18 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 16:44 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 16:44 andrew@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 15:28 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 15:23 andrew@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 15:23 andrew@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 15:23 andrew@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 15:23 andrew@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 15:22 andrew@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 15:22 andrew@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 15:22 andrew@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 15:22 andrew@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 15:22 andrew@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 15:21 andrew@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 15:21 andrew@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
== 2025-01-25 ==
* 16:30 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on netflow1002.eqiad.wmnet with reason: disabling gnmic in systemd
* 15:36 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on netflow1002.eqiad.wmnet with reason: disabling gnmic in systemd
* 11:51 godog: bounce thanos-query on titan100*
* 11:14 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on cr1-magru,cr[1-2]-magru IPv6 with reason: IBGP instability from cr1 to cr2 in magru causing ping faulures from alert1002
* 11:02 topranks: bouncing IBGP session from cr1-magru to cr2-magru manually to reset
* 11:00 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on cr2-magru with reason: IBGP instability from cr1 to cr2 in magru causing ping faulures from alert1002
* 10:34 topranks: cr2-magru: drain eqdfw to magru transport circuit that has flapped a few times overnight
* 10:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2238 (T384592)', diff saved to https://phabricator.wikimedia.org/P72435 and previous config saved to /var/cache/conftool/dbconfig/20250125-100422-marostegui.json
* 09:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P72434 and previous config saved to /var/cache/conftool/dbconfig/20250125-094915-marostegui.json
* 09:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P72433 and previous config saved to /var/cache/conftool/dbconfig/20250125-093408-marostegui.json
* 09:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2238 (T384592)', diff saved to https://phabricator.wikimedia.org/P72432 and previous config saved to /var/cache/conftool/dbconfig/20250125-091901-marostegui.json
* 08:34 fabfur@cumin1002: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool site magru [reason: depool magru to check for cr issues, no task ID specified]
* 08:34 fabfur@cumin1002: START - Cookbook sre.dns.admin DNS admin: depool site magru [reason: depool magru to check for cr issues, no task ID specified]
* 08:31 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2238 (T384592)', diff saved to https://phabricator.wikimedia.org/P72431 and previous config saved to /var/cache/conftool/dbconfig/20250125-083104-marostegui.json
* 08:30 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2238.codfw.wmnet with reason: Maintenance
* 08:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2226 (T384592)', diff saved to https://phabricator.wikimedia.org/P72430 and previous config saved to /var/cache/conftool/dbconfig/20250125-083042-marostegui.json
* 08:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P72429 and previous config saved to /var/cache/conftool/dbconfig/20250125-081535-marostegui.json
* 08:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P72428 and previous config saved to /var/cache/conftool/dbconfig/20250125-080028-marostegui.json
* 07:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2226 (T384592)', diff saved to https://phabricator.wikimedia.org/P72427 and previous config saved to /var/cache/conftool/dbconfig/20250125-074521-marostegui.json
* 07:31 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2226 (T384592)', diff saved to https://phabricator.wikimedia.org/P72426 and previous config saved to /var/cache/conftool/dbconfig/20250125-073059-marostegui.json
* 07:30 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2187.codfw.wmnet with reason: Maintenance
* 07:30 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2226.codfw.wmnet with reason: Maintenance
* 07:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2225 (T384592)', diff saved to https://phabricator.wikimedia.org/P72425 and previous config saved to /var/cache/conftool/dbconfig/20250125-073021-marostegui.json
* 07:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P72424 and previous config saved to /var/cache/conftool/dbconfig/20250125-071514-marostegui.json
* 07:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P72423 and previous config saved to /var/cache/conftool/dbconfig/20250125-070007-marostegui.json
* 06:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2225 (T384592)', diff saved to https://phabricator.wikimedia.org/P72422 and previous config saved to /var/cache/conftool/dbconfig/20250125-064500-marostegui.json
* 05:59 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2225 (T384592)', diff saved to https://phabricator.wikimedia.org/P72421 and previous config saved to /var/cache/conftool/dbconfig/20250125-055917-marostegui.json
* 05:59 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2225.codfw.wmnet with reason: Maintenance
* 05:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T384592)', diff saved to https://phabricator.wikimedia.org/P72420 and previous config saved to /var/cache/conftool/dbconfig/20250125-055855-marostegui.json
* 05:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P72419 and previous config saved to /var/cache/conftool/dbconfig/20250125-054347-marostegui.json
* 05:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P72418 and previous config saved to /var/cache/conftool/dbconfig/20250125-052839-marostegui.json
* 05:13 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T384592)', diff saved to https://phabricator.wikimedia.org/P72417 and previous config saved to /var/cache/conftool/dbconfig/20250125-051332-marostegui.json
* 04:27 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2207 (T384592)', diff saved to https://phabricator.wikimedia.org/P72416 and previous config saved to /var/cache/conftool/dbconfig/20250125-042719-marostegui.json
* 04:27 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2207.codfw.wmnet with reason: Maintenance
* 03:34 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 03:30 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2197.codfw.wmnet with reason: Maintenance
* 03:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T384592)', diff saved to https://phabricator.wikimedia.org/P72415 and previous config saved to /var/cache/conftool/dbconfig/20250125-033035-marostegui.json
* 03:27 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 03:27 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 03:21 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 03:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P72414 and previous config saved to /var/cache/conftool/dbconfig/20250125-031528-marostegui.json
* 03:12 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 03:04 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 03:04 andrew@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 03:04 andrew@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 03:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P72413 and previous config saved to /var/cache/conftool/dbconfig/20250125-030021-marostegui.json
* 02:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T384592)', diff saved to https://phabricator.wikimedia.org/P72412 and previous config saved to /var/cache/conftool/dbconfig/20250125-024514-marostegui.json
* 01:57 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2189 (T384592)', diff saved to https://phabricator.wikimedia.org/P72411 and previous config saved to /var/cache/conftool/dbconfig/20250125-015731-marostegui.json
* 01:57 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2189.codfw.wmnet with reason: Maintenance
* 01:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T384592)', diff saved to https://phabricator.wikimedia.org/P72410 and previous config saved to /var/cache/conftool/dbconfig/20250125-015709-marostegui.json
* 01:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P72409 and previous config saved to /var/cache/conftool/dbconfig/20250125-014201-marostegui.json
* 01:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P72408 and previous config saved to /var/cache/conftool/dbconfig/20250125-012654-marostegui.json
* 01:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T384592)', diff saved to https://phabricator.wikimedia.org/P72407 and previous config saved to /var/cache/conftool/dbconfig/20250125-011147-marostegui.json
* 00:19 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2175 (T384592)', diff saved to https://phabricator.wikimedia.org/P72406 and previous config saved to /var/cache/conftool/dbconfig/20250125-001950-marostegui.json
* 00:19 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2175.codfw.wmnet with reason: Maintenance
* 00:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T384592)', diff saved to https://phabricator.wikimedia.org/P72405 and previous config saved to /var/cache/conftool/dbconfig/20250125-001929-marostegui.json
* 00:08 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudgw1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 00:04 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudgw1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 00:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P72404 and previous config saved to /var/cache/conftool/dbconfig/20250125-000422-marostegui.json
* 00:04 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudgw1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
== 2025-01-24 ==
* 23:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P72403 and previous config saved to /var/cache/conftool/dbconfig/20250124-234914-marostegui.json
* 23:39 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7004.magru.wmnet
* 23:39 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for cp7004.magru.wmnet
* 23:39 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudgw1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 23:36 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7004.magru.wmnet
* 23:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T384592)', diff saved to https://phabricator.wikimedia.org/P72402 and previous config saved to /var/cache/conftool/dbconfig/20250124-233407-marostegui.json
* 23:30 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudgw1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 23:26 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudgw1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:56 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudgw1004
* 22:55 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudgw1004
* 22:54 vriley@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudgw1003
* 22:52 vriley@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudgw1003
* 22:43 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2148 (T384592)', diff saved to https://phabricator.wikimedia.org/P72401 and previous config saved to /var/cache/conftool/dbconfig/20250124-224303-marostegui.json
* 22:42 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2148.codfw.wmnet with reason: Maintenance
* 22:18 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp7004.magru.wmnet
* 22:18 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp7004.magru.wmnet,service=cdn
* 22:11 sukhe: pool bunch of cp7x in magru for ats-be that were depooled
* 22:11 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7015.magru.wmnet,service=(cdnats-be)
* 22:11 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7010.magru.wmnet,service=(cdnats-be)
* 22:11 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp700[2-4].magru.wmnet,service=(cdnats-be)
* 22:10 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7006.magru.wmnet,service=(cdnats-be)
* 22:10 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7008.magru.wmnet,service=(cdnats-be)
* 22:10 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7003.magru.wmnet,service=(cdnats-be)
* 22:10 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7001.magru.wmnet,service=(cdnats-be)
* 22:08 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudgw1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:08 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudgw1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 22:07 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 22:07 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt cloudgw1003 - vriley@cumin1002"
* 22:05 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt cloudgw1003 - vriley@cumin1002"
* 22:02 vriley@cumin1002: START - Cookbook sre.dns.netbox
* 21:51 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7004.magru.wmnet with reason: Thermal settings testing (T373993)
* 21:50 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 21:50 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1246 (T384592)', diff saved to https://phabricator.wikimedia.org/P72399 and previous config saved to /var/cache/conftool/dbconfig/20250124-215037-marostegui.json
* 21:49 brett@puppetserver1001: conftool action : set/pooled=no; selector: name=cp7004.magru.wmnet,service=cdn
* 21:47 brett: Testing thermal settings on cp7004 (T373993)
* 21:43 amastilovic@deploy2002: Finished deploy [airflow-dags/platform_eng@ebb3680]: (no justification provided) (duration: 00m 31s)
* 21:42 amastilovic@deploy2002: Started deploy [airflow-dags/platform_eng@ebb3680]: (no justification provided)
* 21:35 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1246', diff saved to https://phabricator.wikimedia.org/P72398 and previous config saved to /var/cache/conftool/dbconfig/20250124-213530-marostegui.json
* 21:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1246', diff saved to https://phabricator.wikimedia.org/P72397 and previous config saved to /var/cache/conftool/dbconfig/20250124-212023-marostegui.json
* 21:15 amastilovic@deploy2002: Finished deploy [airflow-dags/platform_eng@3907ed7]: (no justification provided) (duration: 00m 10s)
* 21:15 amastilovic@deploy2002: Started deploy [airflow-dags/platform_eng@3907ed7]: (no justification provided)
* 21:13 amastilovic@deploy2002: Finished deploy [airflow-dags/platform_eng@3907ed7]: (no justification provided) (duration: 00m 35s)
* 21:12 amastilovic@deploy2002: Started deploy [airflow-dags/platform_eng@3907ed7]: (no justification provided)
* 21:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1246 (T384592)', diff saved to https://phabricator.wikimedia.org/P72396 and previous config saved to /var/cache/conftool/dbconfig/20250124-210515-marostegui.json
* 20:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P72393 and previous config saved to /var/cache/conftool/dbconfig/20250124-200419-marostegui.json
* 19:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P72392 and previous config saved to /var/cache/conftool/dbconfig/20250124-194911-marostegui.json
* 19:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233 (T384592)', diff saved to https://phabricator.wikimedia.org/P72391 and previous config saved to /var/cache/conftool/dbconfig/20250124-193404-marostegui.json
* 19:27 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 90 days, 0:00:00 on moscovium.eqiad.wmnet with reason: to be decomed
* 19:19 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 90 days, 0:00:00 on moscovium.eqiad.wmnet with reason: to be decomed
* 18:48 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1233 (T384592)', diff saved to https://phabricator.wikimedia.org/P72390 and previous config saved to /var/cache/conftool/dbconfig/20250124-184807-marostegui.json
* 18:48 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1233.eqiad.wmnet with reason: Maintenance
* 18:05 marostegui@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts db2136.codfw.wmnet
* 18:05 marostegui@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 18:05 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1154-1158].eqiad.wmnet
* 18:05 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1154-1158].eqiad.wmnet
* 18:04 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1225.eqiad.wmnet with reason: Maintenance
* 18:03 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 17:59 marostegui: Removing db2136 from zarcillo T384479
* 17:57 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2136.codfw.wmnet
* 17:48 mforns@deploy2002: Finished deploy [airflow-dags/analytics@ebb3680]: bump up mediawiki reduced as part of temp accounts deployment (duration: 01m 00s)
* 17:47 mforns@deploy2002: Started deploy [airflow-dags/analytics@ebb3680]: bump up mediawiki reduced as part of temp accounts deployment
* 17:32 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1154.eqiad.wmnet with OS bookworm
* 17:13 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1154.eqiad.wmnet with reason: host reimage
* 17:10 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1154.eqiad.wmnet with reason: host reimage
* 16:57 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1155.eqiad.wmnet with OS bookworm
* 16:54 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1154
* 16:54 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1154
* 16:54 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1154.eqiad.wmnet with OS bookworm
* 16:47 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1157.eqiad.wmnet with OS bookworm
* 16:47 kamila@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1154.eqiad.wmnet with OS bookworm
* 16:43 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1158.eqiad.wmnet with OS bookworm
* 16:43 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2227,2229-2230].codfw.wmnet
* 16:43 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker[2227,2229-2230].codfw.wmnet
* 16:43 jayme@cumin1002: START - Cookbook sre.hosts.remove-downtime for wikikube-worker[2227,2229-2230].codfw.wmnet
* 16:43 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2227,2229-2230].codfw.wmnet
* 16:35 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2227
* 16:35 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2227
* 16:35 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 16:33 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1156.eqiad.wmnet with OS bookworm
* 16:33 pt1979@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudcephosd1013']
* 16:32 pt1979@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013']
* 16:28 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1157.eqiad.wmnet with reason: host reimage
* 16:26 jayme: imported istioctl 1.24.2-1 to bullseye/bookworm-wikimedia T341984
* 16:24 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1158.eqiad.wmnet with reason: host reimage
* 16:24 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2230
* 16:23 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2230
* 16:21 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1157.eqiad.wmnet with reason: host reimage
* 16:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P72388 and previous config saved to /var/cache/conftool/dbconfig/20250124-162144-marostegui.json
* 16:21 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1158.eqiad.wmnet with reason: host reimage
* 16:21 andrew@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 16:21 andrew@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 16:20 andrew@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 16:20 andrew@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 16:16 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2229
* 16:16 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2229
* 16:13 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1156.eqiad.wmnet with reason: host reimage
* 16:09 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1155.eqiad.wmnet with reason: host reimage
* 16:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P72387 and previous config saved to /var/cache/conftool/dbconfig/20250124-160637-marostegui.json
* 16:06 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1156.eqiad.wmnet with reason: host reimage
* 16:05 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1155.eqiad.wmnet with reason: host reimage
* 16:04 pt1979@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1013']
* 16:04 pt1979@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013']
* 16:00 pt1979@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 15:59 pt1979@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 15:57 andrew@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 15:57 andrew@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 15:57 andrew@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 15:57 andrew@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 15:56 andrew@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 15:56 andrew@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 15:56 andrew@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 15:55 andrew@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 15:52 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1158
* 15:52 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1158
* 15:52 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1158.eqiad.wmnet with OS bookworm
* 15:52 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1157
* 15:52 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1157
* 15:52 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1157.eqiad.wmnet with OS bookworm
* 15:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229 (T384592)', diff saved to https://phabricator.wikimedia.org/P72386 and previous config saved to /var/cache/conftool/dbconfig/20250124-155130-marostegui.json
* 15:50 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1156
* 15:50 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1156
* 15:50 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1156.eqiad.wmnet with OS bookworm
* 15:49 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1155
* 15:49 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1155
* 15:49 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1155.eqiad.wmnet with OS bookworm
* 15:49 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1154
* 15:49 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1154
* 15:49 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1154.eqiad.wmnet with OS bookworm
* 15:47 andrew@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 15:46 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1154.eqiad.wmnet wikikube-worker1155.eqiad.wmnet wikikube-worker1156.eqiad.wmnet wikikube-worker1157.eqiad.wmnet wikikube-worker1158.eqiad.wmnet on all recursors
* 15:46 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1154.eqiad.wmnet wikikube-worker1155.eqiad.wmnet wikikube-worker1156.eqiad.wmnet wikikube-worker1157.eqiad.wmnet wikikube-worker1158.eqiad.wmnet on all recursors
* 15:46 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1017 to wikikube-worker1158
* 15:46 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1158
* 15:44 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1158
* 15:44 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:44 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1017 to wikikube-worker1158 - kamila@cumin1002"
* 15:44 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1017 to wikikube-worker1158 - kamila@cumin1002"
* 15:43 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Slavina Stefanova out of all services on: 1221 hosts
* 15:42 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1016 to wikikube-worker1157
* 15:42 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1157
* 15:42 root@cumin2002: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Slavina Stefanova out of all services on: 1010 hosts
* 15:41 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 15:41 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1157
* 15:41 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:41 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1016 to wikikube-worker1157 - kamila@cumin1002"
* 15:40 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1016 to wikikube-worker1157 - kamila@cumin1002"
* 15:40 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1017 to wikikube-worker1158
* 15:40 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1015 to wikikube-worker1156
* 15:39 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1156
* 15:39 andrew@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 15:38 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 15:38 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1229 (T384592)', diff saved to https://phabricator.wikimedia.org/P72385 and previous config saved to /var/cache/conftool/dbconfig/20250124-153805-marostegui.json
* 15:37 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1229.eqiad.wmnet with reason: Maintenance
* 15:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1222 (T384592)', diff saved to https://phabricator.wikimedia.org/P72384 and previous config saved to /var/cache/conftool/dbconfig/20250124-153743-marostegui.json
* 15:36 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 15:36 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1156
* 15:36 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:36 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1015 to wikikube-worker1156 - kamila@cumin1002"
* 15:36 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1016 to wikikube-worker1157
* 15:33 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1015 to wikikube-worker1156 - kamila@cumin1002"
* 15:32 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1014 to wikikube-worker1155
* 15:32 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1155
* 15:29 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 15:29 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2227,2229-2230].codfw.wmnet
* 15:29 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1015 to wikikube-worker1156
* 15:29 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1155
* 15:29 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:29 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1014 to wikikube-worker1155 - kamila@cumin1002"
* 15:29 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1013 to wikikube-worker1154
* 15:29 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1014 to wikikube-worker1155 - kamila@cumin1002"
* 15:28 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1154
* 15:27 jayme@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker[2227,2229-2230].codfw.wmnet with reason: Depooled via sre.k8s.pool-depool-node
* 15:27 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2227,2229-2230].codfw.wmnet
* 15:25 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 15:25 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1154
* 15:25 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:25 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1013 to wikikube-worker1154 - kamila@cumin1002"
* 15:25 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1014 to wikikube-worker1155
* 15:24 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1013 to wikikube-worker1154 - kamila@cumin1002"
* 15:23 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 15:22 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P72383 and previous config saved to /var/cache/conftool/dbconfig/20250124-152236-marostegui.json
* 15:20 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 15:19 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1013 to wikikube-worker1154
* 15:19 andrew@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 15:18 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host parse[1013-1017].eqiad.wmnet
* 15:12 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host parse[1013-1017].eqiad.wmnet
* 15:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P72382 and previous config saved to /var/cache/conftool/dbconfig/20250124-150729-marostegui.json
* 15:02 andrew@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 15:02 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 14:55 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 14:54 andrew@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 14:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1222 (T384592)', diff saved to https://phabricator.wikimedia.org/P72381 and previous config saved to /var/cache/conftool/dbconfig/20250124-145222-marostegui.json
* 14:46 andrew@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1013.eqiad.wmnet']
* 14:45 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1151.eqiad.wmnet with OS bookworm
* 14:43 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1152.eqiad.wmnet with OS bookworm
* 14:40 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 14:36 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1150.eqiad.wmnet with OS bookworm
* 14:33 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1153.eqiad.wmnet with OS bookworm
* 14:29 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1149.eqiad.wmnet with OS bookworm
* 14:25 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1151.eqiad.wmnet with reason: host reimage
* 14:25 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1148.eqiad.wmnet with OS bookworm
* 14:22 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1152.eqiad.wmnet with reason: host reimage
* 14:17 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1150.eqiad.wmnet with reason: host reimage
* 14:13 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1153.eqiad.wmnet with reason: host reimage
* 14:11 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet
* 14:10 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1149.eqiad.wmnet with reason: host reimage
* 14:07 cmooney@cumin1002: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet
* 14:07 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1148.eqiad.wmnet with reason: host reimage
* 14:05 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 14:05 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 14:04 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1151.eqiad.wmnet with reason: host reimage
* 14:04 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1152.eqiad.wmnet with reason: host reimage
* 14:04 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1150.eqiad.wmnet with reason: host reimage
* 14:04 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1153.eqiad.wmnet with reason: host reimage
* 14:04 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1222 (T384592)', diff saved to https://phabricator.wikimedia.org/P72380 and previous config saved to /var/cache/conftool/dbconfig/20250124-140410-marostegui.json
* 14:04 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1149.eqiad.wmnet with reason: host reimage
* 14:04 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1239.eqiad.wmnet with reason: Maintenance
* 14:03 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1222.eqiad.wmnet with reason: Maintenance
* 14:03 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1148.eqiad.wmnet with reason: host reimage
* 14:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T384592)', diff saved to https://phabricator.wikimedia.org/P72379 and previous config saved to /var/cache/conftool/dbconfig/20250124-140333-marostegui.json
* 13:48 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1153
* 13:48 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1153
* 13:48 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1152
* 13:48 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1152
* 13:48 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1153.eqiad.wmnet with OS bookworm
* 13:48 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1152.eqiad.wmnet with OS bookworm
* 13:48 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1151
* 13:48 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1151
* 13:48 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P72378 and previous config saved to /var/cache/conftool/dbconfig/20250124-134826-marostegui.json
* 13:48 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1151.eqiad.wmnet with OS bookworm
* 13:48 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1150
* 13:48 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1150
* 13:48 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1150.eqiad.wmnet with OS bookworm
* 13:48 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1149
* 13:48 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1149
* 13:48 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1148
* 13:48 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1148
* 13:48 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1149.eqiad.wmnet with OS bookworm
* 13:48 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1148.eqiad.wmnet with OS bookworm
* 13:45 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1148.eqiad.wmnet wikikube-worker1149.eqiad.wmnet wikikube-worker1150.eqiad.wmnet wikikube-worker1151.eqiad.wmnet wikikube-worker1152.eqiad.wmnet wikikube-worker1153.eqiad.wmnet on all recursors
* 13:45 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1148.eqiad.wmnet wikikube-worker1149.eqiad.wmnet wikikube-worker1150.eqiad.wmnet wikikube-worker1151.eqiad.wmnet wikikube-worker1152.eqiad.wmnet wikikube-worker1153.eqiad.wmnet on all recursors
* 13:44 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1012 to wikikube-worker1153
* 13:44 cmooney@cumin1002: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 13:44 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1153
* 13:43 cmooney@cumin1002: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 13:43 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1153
* 13:43 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:43 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1012 to wikikube-worker1153 - kamila@cumin1002"
* 13:42 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1012 to wikikube-worker1153 - kamila@cumin1002"
* 13:41 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1013.eqiad.wmnet with OS bullseye
* 13:40 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1011 to wikikube-worker1152
* 13:40 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1152
* 13:39 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 13:39 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1152
* 13:38 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:38 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1011 to wikikube-worker1152 - kamila@cumin1002"
* 13:38 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1012 to wikikube-worker1153
* 13:38 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1011 to wikikube-worker1152 - kamila@cumin1002"
* 13:36 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1010 to wikikube-worker1151
* 13:35 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1151
* 13:34 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 13:34 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1151
* 13:34 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:34 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1010 to wikikube-worker1151 - kamila@cumin1002"
* 13:34 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1011 to wikikube-worker1152
* 13:34 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1010 to wikikube-worker1151 - kamila@cumin1002"
* 13:34 cmooney@cumin1002: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 13:34 cmooney@cumin1002: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 13:33 cmooney@cumin1002: END (ERROR) - Cookbook sre.netbox.update-extras (exit_code=97) rolling restart_daemons on A:netbox-canary
* 13:33 cmooney@cumin1002: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 13:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P72376 and previous config saved to /var/cache/conftool/dbconfig/20250124-133319-marostegui.json
* 13:32 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1009 to wikikube-worker1150
* 13:31 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1150
* 13:30 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1216.eqiad.wmnet with reason: rebuilding tables
* 13:30 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 13:30 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1010 to wikikube-worker1151
* 13:30 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1150
* 13:30 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:30 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1009 to wikikube-worker1150 - kamila@cumin1002"
* 13:29 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1009 to wikikube-worker1150 - kamila@cumin1002"
* 13:27 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1008 to wikikube-worker1149
* 13:27 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1149
* 13:26 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 13:26 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1149
* 13:26 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:26 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1008 to wikikube-worker1149 - kamila@cumin1002"
* 13:25 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1009 to wikikube-worker1150
* 13:25 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1008 to wikikube-worker1149 - kamila@cumin1002"
* 13:23 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1007 to wikikube-worker1148
* 13:23 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1148
* 13:22 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 13:21 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1148
* 13:21 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:21 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1007 to wikikube-worker1148 - kamila@cumin1002"
* 13:21 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1008 to wikikube-worker1149
* 13:21 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1007 to wikikube-worker1148 - kamila@cumin1002"
* 13:18 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T384592)', diff saved to https://phabricator.wikimedia.org/P72375 and previous config saved to /var/cache/conftool/dbconfig/20250124-131812-marostegui.json
* 13:17 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 13:17 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1007 to wikikube-worker1148
* 13:16 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host parse[1007-1012].eqiad.wmnet
* 13:13 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host parse[1007-1012].eqiad.wmnet
* 13:04 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1197 (T384592)', diff saved to https://phabricator.wikimedia.org/P72374 and previous config saved to /var/cache/conftool/dbconfig/20250124-130431-marostegui.json
* 13:04 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1197.eqiad.wmnet with reason: Maintenance
* 13:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T384592)', diff saved to https://phabricator.wikimedia.org/P72373 and previous config saved to /var/cache/conftool/dbconfig/20250124-130409-marostegui.json
* 12:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P72372 and previous config saved to /var/cache/conftool/dbconfig/20250124-124902-marostegui.json
* 12:46 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1113948Revert "Warn if 'preprint', 'dataset', or 'standard' key is missing" (T384661) (duration: 13m 04s)
* 12:40 lucaswerkmeister-wmde@deploy2002: mvolz, lucaswerkmeister-wmde: Continuing with sync
* 12:38 lucaswerkmeister-wmde@deploy2002: mvolz, lucaswerkmeister-wmde: Backport for gerrit:1113948Revert "Warn if 'preprint', 'dataset', or 'standard' key is missing" (T384661) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 12:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P72371 and previous config saved to /var/cache/conftool/dbconfig/20250124-123355-marostegui.json
* 12:33 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1113948Revert "Warn if 'preprint', 'dataset', or 'standard' key is missing" (T384661)
* 12:18 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T384592)', diff saved to https://phabricator.wikimedia.org/P72370 and previous config saved to /var/cache/conftool/dbconfig/20250124-121848-marostegui.json
* 12:04 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1188 (T384592)', diff saved to https://phabricator.wikimedia.org/P72369 and previous config saved to /var/cache/conftool/dbconfig/20250124-120417-marostegui.json
* 12:04 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1188.eqiad.wmnet with reason: Maintenance
* 12:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T384592)', diff saved to https://phabricator.wikimedia.org/P72368 and previous config saved to /var/cache/conftool/dbconfig/20250124-120355-marostegui.json
* 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ml-etcd2001.codfw.wmnet to plain
* 11:48 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P72367 and previous config saved to /var/cache/conftool/dbconfig/20250124-114848-marostegui.json
* 11:48 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ml-etcd2001.codfw.wmnet to plain
* 11:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2020.codfw.wmnet
* 11:44 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2020.codfw.wmnet
* 11:43 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ml-etcd2001.codfw.wmnet to drbd
* 11:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P72366 and previous config saved to /var/cache/conftool/dbconfig/20250124-113341-marostegui.json
* 11:33 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of ml-etcd2001.codfw.wmnet to drbd
* 11:29 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2020.codfw.wmnet
* 11:25 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2020.codfw.wmnet
* 11:18 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T384592)', diff saved to https://phabricator.wikimedia.org/P72365 and previous config saved to /var/cache/conftool/dbconfig/20250124-111834-marostegui.json
* 10:50 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2140 from dbctl T384480', diff saved to https://phabricator.wikimedia.org/P72363 and previous config saved to /var/cache/conftool/dbconfig/20250124-105029-fceratto.json
* 10:21 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1182 (T384592)', diff saved to https://phabricator.wikimedia.org/P72362 and previous config saved to /var/cache/conftool/dbconfig/20250124-102157-marostegui.json
* 10:21 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1182.eqiad.wmnet with reason: Maintenance
* 10:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T384592)', diff saved to https://phabricator.wikimedia.org/P72361 and previous config saved to /var/cache/conftool/dbconfig/20250124-102135-marostegui.json
* 10:13 mnz@deploy2002: Finished deploy [airflow-dags/research@95b14c7]: (no justification provided) (duration: 00m 43s)
* 10:12 mnz@deploy2002: Started deploy [airflow-dags/research@95b14c7]: (no justification provided)
* 10:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P72360 and previous config saved to /var/cache/conftool/dbconfig/20250124-100628-marostegui.json
* 10:01 mnz@deploy2002: Finished deploy [airflow-dags/research@ba61f77]: (no justification provided) (duration: 00m 12s)
* 10:01 mnz@deploy2002: Started deploy [airflow-dags/research@ba61f77]: (no justification provided)
* 09:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P72359 and previous config saved to /var/cache/conftool/dbconfig/20250124-095121-marostegui.json
* 09:43 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on netflow1002.eqiad.wmnet with reason: disabling alerts as I'm running gnmic manually rather than with systemd
* 09:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T384592)', diff saved to https://phabricator.wikimedia.org/P72358 and previous config saved to /var/cache/conftool/dbconfig/20250124-093614-marostegui.json
* 09:21 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2022.codfw.wmnet to cluster codfw and group B
* 09:20 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2022.codfw.wmnet to cluster codfw and group B
* 09:18 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2022.codfw.wmnet
* 09:14 root@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1216.eqiad.wmnet with OS bookworm
* 09:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2022.codfw.wmnet
* 09:05 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2022.codfw.wmnet with OS bookworm
* 08:51 root@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1216.eqiad.wmnet with reason: host reimage
* 08:49 root@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1225.eqiad.wmnet with OS bookworm
* 08:47 root@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1216.eqiad.wmnet with reason: host reimage
* 08:46 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2022.codfw.wmnet with reason: host reimage
* 08:42 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2022.codfw.wmnet with reason: host reimage
* 08:36 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1156 (T384592)', diff saved to https://phabricator.wikimedia.org/P72357 and previous config saved to /var/cache/conftool/dbconfig/20250124-083638-marostegui.json
* 08:36 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 08:36 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1156.eqiad.wmnet with reason: Maintenance
* 08:30 root@cumin1002: START - Cookbook sre.hosts.reimage for host db1216.eqiad.wmnet with OS bookworm
* 08:29 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2214.codfw.wmnet with reason: Maintenance
* 08:25 root@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1225.eqiad.wmnet with reason: host reimage
* 08:21 root@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1225.eqiad.wmnet with reason: host reimage
* 08:18 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1201.eqiad.wmnet with reason: Maintenance
* 08:11 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1216.eqiad.wmnet with reason: os upgrade
* 08:08 marostegui: Remove es1023 from es5 eqiad dbmaint T384679
* 08:08 marostegui@cumin1002: dbctl commit (dc=all): 'Promote es1044 to es5 master', diff saved to https://phabricator.wikimedia.org/P72356 and previous config saved to /var/cache/conftool/dbconfig/20250124-080804-root.json
* 08:04 root@cumin1002: START - Cookbook sre.hosts.reimage for host db1225.eqiad.wmnet with OS bookworm
* 07:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es1022.eqiad.wmnet
* 07:58 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:58 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es1022.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 07:57 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es1022.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 07:54 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 07:53 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2022.codfw.wmnet with OS bookworm
* 07:51 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1225.eqiad.wmnet with reason: os upgrade
* 07:48 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts es1022.eqiad.wmnet
* 06:42 marostegui@cumin1002: dbctl commit (dc=all): 'db2189 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72355 and previous config saved to /var/cache/conftool/dbconfig/20250124-064249-root.json
* 06:35 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72354 and previous config saved to /var/cache/conftool/dbconfig/20250124-063557-root.json
* 06:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2229 (T384592)', diff saved to https://phabricator.wikimedia.org/P72353 and previous config saved to /var/cache/conftool/dbconfig/20250124-062855-marostegui.json
* 06:27 marostegui@cumin1002: dbctl commit (dc=all): 'db2189 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72352 and previous config saved to /var/cache/conftool/dbconfig/20250124-062744-root.json
* 06:20 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72351 and previous config saved to /var/cache/conftool/dbconfig/20250124-062052-root.json
* 06:13 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P72350 and previous config saved to /var/cache/conftool/dbconfig/20250124-061348-marostegui.json
* 06:12 marostegui@cumin1002: dbctl commit (dc=all): 'db2189 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72349 and previous config saved to /var/cache/conftool/dbconfig/20250124-061238-root.json
* 06:05 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72348 and previous config saved to /var/cache/conftool/dbconfig/20250124-060547-root.json
* 05:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P72347 and previous config saved to /var/cache/conftool/dbconfig/20250124-055842-marostegui.json
* 05:57 marostegui@cumin1002: dbctl commit (dc=all): 'db2189 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72346 and previous config saved to /var/cache/conftool/dbconfig/20250124-055733-root.json
* 05:50 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72345 and previous config saved to /var/cache/conftool/dbconfig/20250124-055042-root.json
* 05:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2229 (T384592)', diff saved to https://phabricator.wikimedia.org/P72344 and previous config saved to /var/cache/conftool/dbconfig/20250124-054335-marostegui.json
* 05:42 marostegui@cumin1002: dbctl commit (dc=all): 'db2189 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72343 and previous config saved to /var/cache/conftool/dbconfig/20250124-054227-root.json
* 05:35 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72342 and previous config saved to /var/cache/conftool/dbconfig/20250124-053535-root.json
* 05:15 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2229 (T384592)', diff saved to https://phabricator.wikimedia.org/P72341 and previous config saved to /var/cache/conftool/dbconfig/20250124-051525-marostegui.json
* 05:15 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2229.codfw.wmnet with reason: Maintenance
* 05:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2224 (T384592)', diff saved to https://phabricator.wikimedia.org/P72340 and previous config saved to /var/cache/conftool/dbconfig/20250124-051503-marostegui.json
* 04:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P72339 and previous config saved to /var/cache/conftool/dbconfig/20250124-045955-marostegui.json
* 04:44 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P72338 and previous config saved to /var/cache/conftool/dbconfig/20250124-044449-marostegui.json
* 04:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2224 (T384592)', diff saved to https://phabricator.wikimedia.org/P72337 and previous config saved to /var/cache/conftool/dbconfig/20250124-042942-marostegui.json
* 04:02 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2224 (T384592)', diff saved to https://phabricator.wikimedia.org/P72336 and previous config saved to /var/cache/conftool/dbconfig/20250124-040239-marostegui.json
* 04:02 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2224.codfw.wmnet with reason: Maintenance
* 04:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T384592)', diff saved to https://phabricator.wikimedia.org/P72335 and previous config saved to /var/cache/conftool/dbconfig/20250124-040216-marostegui.json
* 03:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P72334 and previous config saved to /var/cache/conftool/dbconfig/20250124-034709-marostegui.json
* 03:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P72333 and previous config saved to /var/cache/conftool/dbconfig/20250124-033202-marostegui.json
* 03:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T384592)', diff saved to https://phabricator.wikimedia.org/P72332 and previous config saved to /var/cache/conftool/dbconfig/20250124-031655-marostegui.json
* 02:48 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2217 (T384592)', diff saved to https://phabricator.wikimedia.org/P72331 and previous config saved to /var/cache/conftool/dbconfig/20250124-024851-marostegui.json
* 02:48 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2217.codfw.wmnet with reason: Maintenance
* 02:20 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2197.codfw.wmnet with reason: Maintenance
* 02:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T384592)', diff saved to https://phabricator.wikimedia.org/P72330 and previous config saved to /var/cache/conftool/dbconfig/20250124-022005-marostegui.json
* 02:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P72329 and previous config saved to /var/cache/conftool/dbconfig/20250124-020458-marostegui.json
* 01:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P72328 and previous config saved to /var/cache/conftool/dbconfig/20250124-014951-marostegui.json
* 01:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T384592)', diff saved to https://phabricator.wikimedia.org/P72327 and previous config saved to /var/cache/conftool/dbconfig/20250124-013444-marostegui.json
* 01:26 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2193 (T384592)', diff saved to https://phabricator.wikimedia.org/P72326 and previous config saved to /var/cache/conftool/dbconfig/20250124-012645-marostegui.json
* 01:26 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2193.codfw.wmnet with reason: Maintenance
* 01:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T384592)', diff saved to https://phabricator.wikimedia.org/P72325 and previous config saved to /var/cache/conftool/dbconfig/20250124-012623-marostegui.json
* 01:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P72324 and previous config saved to /var/cache/conftool/dbconfig/20250124-011116-marostegui.json
* 00:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P72323 and previous config saved to /var/cache/conftool/dbconfig/20250124-005609-marostegui.json
* 00:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T384592)', diff saved to https://phabricator.wikimedia.org/P72322 and previous config saved to /var/cache/conftool/dbconfig/20250124-004102-marostegui.json
* 00:32 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2180 (T384592)', diff saved to https://phabricator.wikimedia.org/P72321 and previous config saved to /var/cache/conftool/dbconfig/20250124-003237-marostegui.json
* 00:32 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2180.codfw.wmnet with reason: Maintenance
* 00:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169 (T384592)', diff saved to https://phabricator.wikimedia.org/P72320 and previous config saved to /var/cache/conftool/dbconfig/20250124-003215-marostegui.json
* 00:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P72319 and previous config saved to /var/cache/conftool/dbconfig/20250124-001708-marostegui.json
* 00:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P72318 and previous config saved to /var/cache/conftool/dbconfig/20250124-000200-marostegui.json
== 2025-01-23 ==
* 23:46 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169 (T384592)', diff saved to https://phabricator.wikimedia.org/P72317 and previous config saved to /var/cache/conftool/dbconfig/20250123-234653-marostegui.json
* 23:16 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2169 (T384592)', diff saved to https://phabricator.wikimedia.org/P72316 and previous config saved to /var/cache/conftool/dbconfig/20250123-231641-marostegui.json
* 23:16 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2169.codfw.wmnet with reason: Maintenance
* 23:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T384592)', diff saved to https://phabricator.wikimedia.org/P72315 and previous config saved to /var/cache/conftool/dbconfig/20250123-231619-marostegui.json
* 23:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P72312 and previous config saved to /var/cache/conftool/dbconfig/20250123-230112-marostegui.json
* 22:46 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P72311 and previous config saved to /var/cache/conftool/dbconfig/20250123-224604-marostegui.json
* 22:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T384592)', diff saved to https://phabricator.wikimedia.org/P72310 and previous config saved to /var/cache/conftool/dbconfig/20250123-223057-marostegui.json
* 22:13 cmooney@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cloudsw2-d5-eqiad
* 22:12 cmooney@cumin1002: START - Cookbook sre.network.tls for network device cloudsw2-d5-eqiad
* 22:10 cjming: end of UTC late backport window
* 22:09 cjming@deploy2002: Finished scap sync-world: Backport for gerrit:1113834For Parsoid calls, treat preprocessing as starting in SOL state (T382464) (duration: 11m 45s)
* 22:02 cjming@deploy2002: ssastry, cjming: Continuing with sync
* 22:02 cjming@deploy2002: ssastry, cjming: Backport for gerrit:1113834For Parsoid calls, treat preprocessing as starting in SOL state (T382464) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:59 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on netflow1002.eqiad.wmnet with reason: disabling alerts as I'm running gnmic manually rather than with systemd
* 21:58 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2158 (T384592)', diff saved to https://phabricator.wikimedia.org/P72309 and previous config saved to /var/cache/conftool/dbconfig/20250123-215855-marostegui.json
* 21:58 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on db2187.codfw.wmnet with reason: Maintenance
* 21:58 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2158.codfw.wmnet with reason: Maintenance
* 21:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151 (T384592)', diff saved to https://phabricator.wikimedia.org/P72308 and previous config saved to /var/cache/conftool/dbconfig/20250123-215828-marostegui.json
* 21:57 cjming@deploy2002: Started scap sync-world: Backport for gerrit:1113834For Parsoid calls, treat preprocessing as starting in SOL state (T382464)
* 21:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P72307 and previous config saved to /var/cache/conftool/dbconfig/20250123-214320-marostegui.json
* 21:36 cjming@deploy2002: Finished scap sync-world: Backport for gerrit:1081460Profiler: centralize metrics send to a function (duration: 15m 28s)
* 21:29 cjming@deploy2002: cwhite, cjming: Continuing with sync
* 21:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P72306 and previous config saved to /var/cache/conftool/dbconfig/20250123-212813-marostegui.json
* 21:25 cjming@deploy2002: cwhite, cjming: Backport for gerrit:1081460Profiler: centralize metrics send to a function synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:20 cjming@deploy2002: Started scap sync-world: Backport for gerrit:1081460Profiler: centralize metrics send to a function
* 21:14 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on netflow7001.magru.wmnet with reason: disabling alerts as I'm running gnmic manually rather than with systemd
* 21:13 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151 (T384592)', diff saved to https://phabricator.wikimedia.org/P72305 and previous config saved to /var/cache/conftool/dbconfig/20250123-211306-marostegui.json
* 20:42 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2151 (T384592)', diff saved to https://phabricator.wikimedia.org/P72304 and previous config saved to /var/cache/conftool/dbconfig/20250123-204245-marostegui.json
* 20:42 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2151.codfw.wmnet with reason: Maintenance
* 20:12 cdanis@deploy2002: cdanis: Continuing with sync
* 20:11 cdanis@deploy2002: cdanis: Backport for gerrit:1113856tracing: lowercase headers before processing them (T384629) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 20:10 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
* 20:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T384592)', diff saved to https://phabricator.wikimedia.org/P72303 and previous config saved to /var/cache/conftool/dbconfig/20250123-201038-marostegui.json
* 20:07 cdanis@deploy2002: Started scap sync-world: Backport for gerrit:1113856tracing: lowercase headers before processing them (T384629)
* 19:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P72302 and previous config saved to /var/cache/conftool/dbconfig/20250123-195531-marostegui.json
* 19:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P72301 and previous config saved to /var/cache/conftool/dbconfig/20250123-194024-marostegui.json
* 19:33 brennen@deploy2002: rebuilt and synchronized wikiversions files: group2 to 1.44.0-wmf.13 refs T382364
* 19:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T384592)', diff saved to https://phabricator.wikimedia.org/P72300 and previous config saved to /var/cache/conftool/dbconfig/20250123-192517-marostegui.json
* 19:22 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudgw1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 19:22 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudgw1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 19:21 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudgw1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 19:21 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudgw1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 19:19 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudgw1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 19:19 vriley@cumin1002: START - Cookbook sre.hosts.provision for host cloudgw1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
* 19:18 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 19:18 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt clou~dgw1004 - vriley@cumin1002"
* 19:18 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt clou~dgw1004 - vriley@cumin1002"
* 19:18 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1231 (T384592)', diff saved to https://phabricator.wikimedia.org/P72299 and previous config saved to /var/cache/conftool/dbconfig/20250123-191808-marostegui.json
* 19:18 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1231.eqiad.wmnet with reason: Maintenance
* 19:15 brennen: 1.44.0-wmf.13 train (T382364): no current blockers, logs relatively clean, rolling to all wikis.
* 19:14 vriley@cumin1002: START - Cookbook sre.dns.netbox
* 19:14 vriley@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 19:11 vriley@cumin1002: START - Cookbook sre.dns.netbox
* 19:04 taavi: fix my netbox account T373702
* 18:44 cmooney@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr2-eqdfw
* 18:44 cmooney@cumin1002: START - Cookbook sre.network.tls for network device cr2-eqdfw
* 18:43 cmooney@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr2-eqord
* 18:43 cmooney@cumin1002: START - Cookbook sre.network.tls for network device cr2-eqord
* 18:32 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1225.eqiad.wmnet with reason: Maintenance
* 18:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T384592)', diff saved to https://phabricator.wikimedia.org/P72298 and previous config saved to /var/cache/conftool/dbconfig/20250123-183202-marostegui.json
* 18:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P72297 and previous config saved to /var/cache/conftool/dbconfig/20250123-181655-marostegui.json
* 18:09 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1142-1147].eqiad.wmnet
* 18:09 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1142-1147].eqiad.wmnet
* 18:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P72296 and previous config saved to /var/cache/conftool/dbconfig/20250123-180148-marostegui.json
* 17:46 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T384592)', diff saved to https://phabricator.wikimedia.org/P72295 and previous config saved to /var/cache/conftool/dbconfig/20250123-174641-marostegui.json
* 17:38 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1187 (T384592)', diff saved to https://phabricator.wikimedia.org/P72294 and previous config saved to /var/cache/conftool/dbconfig/20250123-173852-marostegui.json
* 17:38 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1187.eqiad.wmnet with reason: Maintenance
* 17:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T384592)', diff saved to https://phabricator.wikimedia.org/P72293 and previous config saved to /var/cache/conftool/dbconfig/20250123-173830-marostegui.json
* 17:33 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cassandra-dev2001
* 17:33 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cassandra-dev2001
* 17:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P72292 and previous config saved to /var/cache/conftool/dbconfig/20250123-172323-marostegui.json
* 17:20 papaul: power down cassandra-dev2001 for maintenance
* 17:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P72290 and previous config saved to /var/cache/conftool/dbconfig/20250123-170816-marostegui.json
* 17:00 elukey@deploy2002: helmfile [staging] DONE helmfile.d/services/kartotherian: sync
* 16:59 elukey@deploy2002: helmfile [staging] START helmfile.d/services/kartotherian: sync
* 16:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T384592)', diff saved to https://phabricator.wikimedia.org/P72289 and previous config saved to /var/cache/conftool/dbconfig/20250123-165309-marostegui.json
* 16:47 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1222.eqiad.wmnet with reason: Index rebuild
* 16:44 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1180 (T384592)', diff saved to https://phabricator.wikimedia.org/P72288 and previous config saved to /var/cache/conftool/dbconfig/20250123-164453-marostegui.json
* 16:44 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1180.eqiad.wmnet with reason: Maintenance
* 16:44 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1173 (T384592)', diff saved to https://phabricator.wikimedia.org/P72287 and previous config saved to /var/cache/conftool/dbconfig/20250123-164431-marostegui.json
* 16:44 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1222 T384597', diff saved to https://phabricator.wikimedia.org/P72286 and previous config saved to /var/cache/conftool/dbconfig/20250123-164415-marostegui.json
* 16:43 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1162 to s2 primary T384597', diff saved to https://phabricator.wikimedia.org/P72285 and previous config saved to /var/cache/conftool/dbconfig/20250123-164322-root.json
* 16:42 marostegui: Starting s2 eqiad failover from db1222 to db1162 - T384597
* 16:40 dcausse@deploy2002: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 16:39 dcausse@deploy2002: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 16:37 dcausse@deploy2002: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 16:37 dcausse@deploy2002: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 16:34 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 16:33 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 16:33 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: demonstration - bking@cumin2002 - T380752
* 16:33 dcausse@deploy2002: helmfile [eqiad] DONE helmfile.d/services/eventstreams-internal: apply
* 16:31 dcausse@deploy2002: helmfile [eqiad] START helmfile.d/services/eventstreams-internal: apply
* 16:29 dcausse@deploy2002: helmfile [codfw] DONE helmfile.d/services/eventstreams-internal: apply
* 16:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P72284 and previous config saved to /var/cache/conftool/dbconfig/20250123-162924-marostegui.json
* 16:28 dcausse@deploy2002: helmfile [codfw] START helmfile.d/services/eventstreams-internal: apply
* 16:26 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s2 T384597
* 16:25 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1162 with weight 0 T384597', diff saved to https://phabricator.wikimedia.org/P72283 and previous config saved to /var/cache/conftool/dbconfig/20250123-162552-root.json
* 16:23 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/eventstreams-internal: apply
* 16:22 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2022.codfw.wmnet with reason: remove from cluster for reimage
* 16:22 marostegui@cumin1002: dbctl commit (dc=all): 'db1162 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72282 and previous config saved to /var/cache/conftool/dbconfig/20250123-162235-root.json
* 16:22 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/eventstreams-internal: apply
* 16:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P72281 and previous config saved to /var/cache/conftool/dbconfig/20250123-161417-marostegui.json
* 16:07 marostegui@cumin1002: dbctl commit (dc=all): 'db1162 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72280 and previous config saved to /var/cache/conftool/dbconfig/20250123-160730-root.json
* 15:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1173 (T384592)', diff saved to https://phabricator.wikimedia.org/P72279 and previous config saved to /var/cache/conftool/dbconfig/20250123-155910-marostegui.json
* 15:56 jgiannelos@deploy2002: helmfile [staging] DONE helmfile.d/services/kartotherian: apply
* 15:55 jgiannelos@deploy2002: helmfile [staging] START helmfile.d/services/kartotherian: apply
* 15:50 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1173 (T384592)', diff saved to https://phabricator.wikimedia.org/P72278 and previous config saved to /var/cache/conftool/dbconfig/20250123-155045-marostegui.json
* 15:50 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1173.eqiad.wmnet with reason: Maintenance
* 15:50 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T384592)', diff saved to https://phabricator.wikimedia.org/P72277 and previous config saved to /var/cache/conftool/dbconfig/20250123-155023-marostegui.json
* 15:50 fceratto@cumin1002: dbctl commit (dc=all): 'Removing db2136 T384479', diff saved to https://phabricator.wikimedia.org/P72276 and previous config saved to /var/cache/conftool/dbconfig/20250123-155016-fceratto.json
* 15:48 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db2189
* 15:48 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host db2189
* 15:36 kamila@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker[1142-1147].eqiad.wmnet
* 15:36 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1142-1147].eqiad.wmnet
* 15:35 brennen@deploy2002: rebuilt and synchronized wikiversions files: group1 to 1.44.0-wmf.13 refs T382364
* 15:35 kamila@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker[1142-1147].eqiad.wmnet
* 15:35 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1142-1147].eqiad.wmnet
* 15:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P72274 and previous config saved to /var/cache/conftool/dbconfig/20250123-153405-marostegui.json
* 15:31 kamila@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker[1142-1147].eqiad.wmnet
* 15:31 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1142-1147].eqiad.wmnet
* 15:23 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1142.eqiad.wmnet with OS bookworm
* 15:21 brennen: 1.44.0-wmf.13 train (T382364): unblocked, rolling to group1
* 15:18 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P72273 and previous config saved to /var/cache/conftool/dbconfig/20250123-151858-marostegui.json
* 15:18 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1147.eqiad.wmnet with OS bookworm
* 15:15 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1143.eqiad.wmnet with OS bookworm
* 15:13 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1146.eqiad.wmnet with OS bookworm
* 15:11 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1145.eqiad.wmnet with OS bookworm
* 15:06 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1144.eqiad.wmnet with OS bookworm
* 15:04 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1142.eqiad.wmnet with reason: host reimage
* 15:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T384592)', diff saved to https://phabricator.wikimedia.org/P72272 and previous config saved to /var/cache/conftool/dbconfig/20250123-150351-marostegui.json
* 14:59 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1147.eqiad.wmnet with reason: host reimage
* 14:56 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1143.eqiad.wmnet with reason: host reimage
* 14:55 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1168 (T384592)', diff saved to https://phabricator.wikimedia.org/P72271 and previous config saved to /var/cache/conftool/dbconfig/20250123-145540-marostegui.json
* 14:55 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1168.eqiad.wmnet with reason: Maintenance
* 14:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T384592)', diff saved to https://phabricator.wikimedia.org/P72270 and previous config saved to /var/cache/conftool/dbconfig/20250123-145518-marostegui.json
* 14:53 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1146.eqiad.wmnet with reason: host reimage
* 14:53 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: demonstration - bking@cumin2002 - T380752
* 14:50 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1145.eqiad.wmnet with reason: host reimage
* 14:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2022.codfw.wmnet
* 14:46 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1144.eqiad.wmnet with reason: host reimage
* 14:43 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1147.eqiad.wmnet with reason: host reimage
* 14:43 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1146.eqiad.wmnet with reason: host reimage
* 14:43 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1145.eqiad.wmnet with reason: host reimage
* 14:43 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1143.eqiad.wmnet with reason: host reimage
* 14:42 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1142.eqiad.wmnet with reason: host reimage
* 14:42 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1144.eqiad.wmnet with reason: host reimage
* 14:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P72269 and previous config saved to /var/cache/conftool/dbconfig/20250123-144011-marostegui.json
* 14:39 vgutierrez: updating acme-chief on acmechief1002
* 14:33 root@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1239.eqiad.wmnet with OS bookworm
* 14:28 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1147
* 14:28 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1147
* 14:28 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1147.eqiad.wmnet with OS bookworm
* 14:28 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1146
* 14:27 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1146
* 14:27 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1146.eqiad.wmnet with OS bookworm
* 14:27 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1145
* 14:27 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1145
* 14:27 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1145.eqiad.wmnet with OS bookworm
* 14:27 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1144
* 14:27 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1144
* 14:27 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1144.eqiad.wmnet with OS bookworm
* 14:27 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1143
* 14:27 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1143
* 14:27 marostegui@cumin1002: dbctl commit (dc=all): 'db2166 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72267 and previous config saved to /var/cache/conftool/dbconfig/20250123-142715-root.json
* 14:27 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1143.eqiad.wmnet with OS bookworm
* 14:27 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1142
* 14:27 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1142
* 14:27 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1142.eqiad.wmnet with OS bookworm
* 14:26 TheresNoTime: UTC afternoon backport window done
* 14:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P72266 and previous config saved to /var/cache/conftool/dbconfig/20250123-142504-marostegui.json
* 14:21 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1142.eqiad.wmnet wikikube-worker1143.eqiad.wmnet wikikube-worker1144.eqiad.wmnet wikikube-worker1145.eqiad.wmnet wikikube-worker1146.eqiad.wmnet wikikube-worker1147.eqiad.wmnet on all recursors
* 14:21 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1142.eqiad.wmnet wikikube-worker1143.eqiad.wmnet wikikube-worker1144.eqiad.wmnet wikikube-worker1145.eqiad.wmnet wikikube-worker1146.eqiad.wmnet wikikube-worker1147.eqiad.wmnet on all recursors
* 14:21 samtar@deploy2002: Finished scap sync-world: Backport for gerrit:1113463cirrus: stop writing to wikitech index from the MW JobQueue, gerrit:1113750cirrus: cleanup unused settings (T374702) (duration: 12m 00s)
* 14:14 samtar@deploy2002: dcausse, samtar: Continuing with sync
* 14:13 samtar@deploy2002: dcausse, samtar: Backport for gerrit:1113463cirrus: stop writing to wikitech index from the MW JobQueue, gerrit:1113750cirrus: cleanup unused settings (T374702) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:12 marostegui@cumin1002: dbctl commit (dc=all): 'db2166 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72264 and previous config saved to /var/cache/conftool/dbconfig/20250123-141209-root.json
* 14:10 root@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1239.eqiad.wmnet with reason: host reimage
* 14:09 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T384592)', diff saved to https://phabricator.wikimedia.org/P72263 and previous config saved to /var/cache/conftool/dbconfig/20250123-140957-marostegui.json
* 14:09 samtar@deploy2002: Started scap sync-world: Backport for gerrit:1113463cirrus: stop writing to wikitech index from the MW JobQueue, gerrit:1113750cirrus: cleanup unused settings (T374702)
* 14:07 root@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1239.eqiad.wmnet with reason: host reimage
* 14:07 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-restart-tcp-mss-clamper (exit_code=0) rolling restart_daemons on A:cp-text_eqiad
* 14:06 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1165 (T384592)', diff saved to https://phabricator.wikimedia.org/P72262 and previous config saved to /var/cache/conftool/dbconfig/20250123-140649-marostegui.json
* 14:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 14:06 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1165.eqiad.wmnet with reason: Maintenance
* 14:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repool db1165', diff saved to https://phabricator.wikimedia.org/P72261 and previous config saved to /var/cache/conftool/dbconfig/20250123-140524-marostegui.json
* 13:57 marostegui@cumin1002: dbctl commit (dc=all): 'db2166 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72259 and previous config saved to /var/cache/conftool/dbconfig/20250123-135704-root.json
* 13:57 marostegui@cumin1002: dbctl commit (dc=all): 'db1165 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72258 and previous config saved to /var/cache/conftool/dbconfig/20250123-135704-root.json
* 13:56 fceratto@cumin1002: dbctl commit (dc=all): 'Depool db2140 T384480', diff saved to https://phabricator.wikimedia.org/P72257 and previous config saved to /var/cache/conftool/dbconfig/20250123-135655-fceratto.json
* 13:55 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1006 to wikikube-worker1147
* 13:54 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1147
* 13:52 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1147
* 13:52 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:52 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1006 to wikikube-worker1147 - kamila@cumin1002"
* 13:50 root@cumin1002: START - Cookbook sre.hosts.reimage for host db1239.eqiad.wmnet with OS bookworm
* 13:50 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1006 to wikikube-worker1147 - kamila@cumin1002"
* 13:49 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-restart-tcp-mss-clamper rolling restart_daemons on A:cp-text_eqiad
* 13:49 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-restart-tcp-mss-clamper (exit_code=0) rolling restart_daemons on A:cp-text_esams
* 13:46 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1239.eqiad.wmnet with reason: reimage
* 13:42 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1005 to wikikube-worker1146
* 13:42 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1146
* 13:41 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 13:41 godog: bounce mtail on centrallog2002 - high system cpu usage and perf top reports native_queued_spin_lock_slowpath
* 13:41 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1006 to wikikube-worker1147
* 13:38 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1146
* 13:38 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:38 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1005 to wikikube-worker1146 - kamila@cumin1002"
* 13:37 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1005 to wikikube-worker1146 - kamila@cumin1002"
* 13:33 marostegui@cumin1002: dbctl commit (dc=all): 'db2166 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72256 and previous config saved to /var/cache/conftool/dbconfig/20250123-133311-root.json
* 13:33 marostegui@cumin1002: dbctl commit (dc=all): 'db1165 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72255 and previous config saved to /var/cache/conftool/dbconfig/20250123-133304-root.json
* 13:31 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-restart-tcp-mss-clamper rolling restart_daemons on A:cp-text_esams
* 13:31 ladsgroup@deploy2002: Finished scap sync-world: Backport for gerrit:1113799file: Add caller to write queries (T384481) (duration: 09m 43s)
* 13:30 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1004 to wikikube-worker1145
* 13:30 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1145
* 13:29 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on netflow1002.eqiad.wmnet with reason: disabling alerts as I'm running gnmic manually rather than with systemd
* 13:29 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 13:28 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1145
* 13:28 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:28 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1004 to wikikube-worker1145 - kamila@cumin1002"
* 13:28 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1005 to wikikube-worker1146
* 13:28 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1004 to wikikube-worker1145 - kamila@cumin1002"
* 13:25 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1003 to wikikube-worker1144
* 13:24 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1144
* 13:24 ladsgroup@deploy2002: ladsgroup: Continuing with sync
* 13:24 ladsgroup@deploy2002: ladsgroup: Backport for gerrit:1113799file: Add caller to write queries (T384481) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 13:23 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 13:23 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1144
* 13:23 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:23 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1003 to wikikube-worker1144 - kamila@cumin1002"
* 13:23 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1004 to wikikube-worker1145
* 13:23 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1003 to wikikube-worker1144 - kamila@cumin1002"
* 13:21 ladsgroup@deploy2002: Started scap sync-world: Backport for gerrit:1113799file: Add caller to write queries (T384481)
* 13:20 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1002 to wikikube-worker1143
* 13:20 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1143
* 13:19 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 13:19 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1143
* 13:19 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:18 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1002 to wikikube-worker1143 - kamila@cumin1002"
* 13:18 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1003 to wikikube-worker1144
* 13:18 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1002 to wikikube-worker1143 - kamila@cumin1002"
* 13:18 marostegui@cumin1002: dbctl commit (dc=all): 'db2166 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72253 and previous config saved to /var/cache/conftool/dbconfig/20250123-131805-root.json
* 13:17 marostegui@cumin1002: dbctl commit (dc=all): 'db1165 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72252 and previous config saved to /var/cache/conftool/dbconfig/20250123-131758-root.json
* 13:17 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1162.eqiad.wmnet with reason: Index rebuild
* 13:16 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from parse1001 to wikikube-worker1142
* 13:16 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1142
* 13:15 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 13:15 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1142
* 13:14 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:14 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1001 to wikikube-worker1142 - kamila@cumin1002"
* 13:14 dcausse@deploy2002: helmfile [eqiad] DONE helmfile.d/services/rdf-streaming-updater: apply
* 13:14 cmooney@cumin1002: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device fasw2-c1a-eqiad
* 13:14 cmooney@cumin1002: START - Cookbook sre.network.tls for network device fasw2-c1a-eqiad
* 13:14 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1002 to wikikube-worker1143
* 13:14 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming parse1001 to wikikube-worker1142 - kamila@cumin1002"
* 13:14 dcausse@deploy2002: helmfile [eqiad] START helmfile.d/services/rdf-streaming-updater: apply
* 13:11 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 13:10 kamila@cumin1002: START - Cookbook sre.hosts.rename from parse1001 to wikikube-worker1142
* 13:09 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host parse[1001-1006].eqiad.wmnet
* 13:09 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1162 T384597', diff saved to https://phabricator.wikimedia.org/P72251 and previous config saved to /var/cache/conftool/dbconfig/20250123-130937-marostegui.json
* 13:07 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Index
* 13:06 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host parse[1001-1006].eqiad.wmnet
* 13:02 marostegui@cumin1002: dbctl commit (dc=all): 'db1165 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72250 and previous config saved to /var/cache/conftool/dbconfig/20250123-130253-root.json
* 12:51 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on netflow1002.eqiad.wmnet with reason: disabling alerts as I'm running gnmic manually rather than with systemd
* 12:41 topranks: restarting gnmic.service on netflow1002
* 12:37 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1165 (T384592)', diff saved to https://phabricator.wikimedia.org/P72249 and previous config saved to /var/cache/conftool/dbconfig/20250123-123708-marostegui.json
* 12:37 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
* 12:36 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1165.eqiad.wmnet with reason: Maintenance
* 12:31 marostegui: Deploy schema change on s8 codfw with replication dbmaint T384592
* 12:23 hnowlan@deploy2002: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 12:23 hnowlan@deploy2002: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 12:21 hnowlan@deploy2002: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 12:21 hnowlan@deploy2002: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 12:17 federico3: Removing db2134 from zarcillo T384476
* 12:13 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-restart-tcp-mss-clamper (exit_code=0) rolling restart_daemons on A:cp-text_drmrs
* 12:13 fceratto@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2134.codfw.wmnet
* 12:13 fceratto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:13 fceratto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2134.codfw.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1002"
* 12:12 fceratto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2134.codfw.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1002"
* 12:05 fceratto@cumin1002: START - Cookbook sre.dns.netbox
* 12:00 fceratto@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2134.codfw.wmnet
* 11:53 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-restart-tcp-mss-clamper rolling restart_daemons on A:cp-text_drmrs
* 11:52 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-restart-tcp-mss-clamper (exit_code=0) rolling restart_daemons on A:cp-text_codfw
* 11:51 urbanecm@deploy2002: Finished scap sync-world: Backport for gerrit:1113761Remove GEInfoboxTemplatesTest (T384254), gerrit:1113760Remove GEInfoboxTemplatesTest (T384254) (duration: 15m 23s)
* 11:49 root@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1204.eqiad.wmnet with OS bookworm
* 11:37 vgutierrez: upload acme-chief 0.38 to apt.wm.org (bookworm-wikimedia)
* 11:35 urbanecm@deploy2002: Started scap sync-world: Backport for gerrit:1113761Remove GEInfoboxTemplatesTest (T384254), gerrit:1113760Remove GEInfoboxTemplatesTest (T384254)
* 11:35 urbanecm@deploy2002: Sync cancelled.
* 11:34 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-restart-tcp-mss-clamper rolling restart_daemons on A:cp-text_codfw
* 11:34 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2022.codfw.wmnet
* 11:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2022.codfw.wmnet
* 11:31 fceratto@cumin1002: dbctl commit (dc=all): 'Depool db2136 T384479', diff saved to https://phabricator.wikimedia.org/P72247 and previous config saved to /var/cache/conftool/dbconfig/20250123-113157-fceratto.json
* 11:28 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-restart-tcp-mss-clamper (exit_code=0) rolling restart_daemons on A:cp-text_eqsin
* 11:28 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2022.codfw.wmnet
* 11:26 root@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1204.eqiad.wmnet with reason: host reimage
* 11:23 root@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1204.eqiad.wmnet with reason: host reimage
* 11:19 urbanecm@deploy2002: urbanecm: Backport for gerrit:1113761Remove GEInfoboxTemplatesTest (T384254), gerrit:1113760Remove GEInfoboxTemplatesTest (T384254) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 11:18 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply
* 11:17 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply
* 11:14 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2032.codfw.wmnet to cluster codfw and group B
* 11:14 urbanecm@deploy2002: Started scap sync-world: Backport for gerrit:1113761Remove GEInfoboxTemplatesTest (T384254), gerrit:1113760Remove GEInfoboxTemplatesTest (T384254)
* 11:14 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2032.codfw.wmnet to cluster codfw and group B
* 11:12 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2032.codfw.wmnet
* 11:08 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-restart-tcp-mss-clamper rolling restart_daemons on A:cp-text_eqsin
* 11:06 root@cumin1002: START - Cookbook sre.hosts.reimage for host db1204.eqiad.wmnet with OS bookworm
* 11:04 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1205.eqiad.wmnet with reason: os upgrade
* 11:04 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2032.codfw.wmnet
* 11:04 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1204.eqiad.wmnet with reason: os upgrade
* 10:57 jynus: pausing media backups on eqiad for maintenance T383902
* 10:56 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2032.codfw.wmnet with OS bookworm
* 10:46 jelto@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab to new version
* 10:36 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2032.codfw.wmnet with reason: host reimage
* 10:32 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2032.codfw.wmnet with reason: host reimage
* 10:26 mvernon@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on ms-be2075.codfw.wmnet with reason: hardware broken awaiting vendor action
* 10:24 dcausse@deploy2002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 10:24 dcausse@deploy2002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 10:22 dcausse@deploy2002: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 10:19 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-restart-tcp-mss-clamper (exit_code=0) rolling restart_daemons on A:cp-text_magru
* 10:18 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_esams and A:cp
* 10:14 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2032.codfw.wmnet with OS bookworm
* 10:10 dcausse@deploy2002: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
* 10:08 moritzm: installing routinator security updates
* 10:05 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 10:05 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
* 10:04 btullis@dns1004: END - running authdns-update
* 10:02 btullis@dns1004: START - running authdns-update
* 10:01 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-restart-tcp-mss-clamper rolling restart_daemons on A:cp-text_magru
* 09:55 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2032.codfw.wmnet with reason: remove from cluster for reimage
* 09:54 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-restart-tcp-mss-clamper (exit_code=0) rolling restart_daemons on A:cp-text_ulsfo
* 09:53 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_esams and A:cp
* 09:53 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_esams and A:cp
* 09:51 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 09:51 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
* 09:45 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2032.codfw.wmnet
* 09:35 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-restart-tcp-mss-clamper rolling restart_daemons on A:cp-text_ulsfo
* 09:30 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_esams and A:cp
* 09:22 dcausse@deploy2002: helmfile [codfw] DONE helmfile.d/services/rdf-streaming-updater: apply
* 09:22 dcausse@deploy2002: helmfile [codfw] START helmfile.d/services/rdf-streaming-updater: apply
* 08:57 dcausse@deploy2002: helmfile [eqiad] DONE helmfile.d/services/rdf-streaming-updater: apply
* 08:57 dcausse@deploy2002: helmfile [eqiad] START helmfile.d/services/rdf-streaming-updater: apply
* 08:53 dcausse@deploy2002: helmfile [codfw] DONE helmfile.d/services/rdf-streaming-updater: apply
* 08:52 dcausse@deploy2002: helmfile [codfw] START helmfile.d/services/rdf-streaming-updater: apply
* 08:49 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 08:48 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 08:35 marostegui@cumin1002: dbctl commit (dc=all): 'Repool pc2 after kernel reboots', diff saved to https://phabricator.wikimedia.org/P72244 and previous config saved to /var/cache/conftool/dbconfig/20250123-083524-marostegui.json
* 08:26 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc2012.codfw.wmnet with reason: Kernel reboot
* 08:26 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc1012.eqiad.wmnet with reason: Kernel reboot
* 08:25 marostegui@cumin1002: dbctl commit (dc=all): 'Depool pc2 for kernel reboots', diff saved to https://phabricator.wikimedia.org/P72242 and previous config saved to /var/cache/conftool/dbconfig/20250123-082545-marostegui.json
* 08:03 moritzm: installing glibc updates on bullseye
* 07:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repool pc1 after kernel reboots', diff saved to https://phabricator.wikimedia.org/P72241 and previous config saved to /var/cache/conftool/dbconfig/20250123-074759-marostegui.json
* 07:36 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc1011.eqiad.wmnet with reason: Kernel reboot
* 07:36 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc2011.codfw.wmnet with reason: Kernel reboot
* 07:35 marostegui@cumin1002: dbctl commit (dc=all): 'Depool pc1 for kernel reboots', diff saved to https://phabricator.wikimedia.org/P72240 and previous config saved to /var/cache/conftool/dbconfig/20250123-073557-marostegui.json
* 07:29 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab to new version
* 07:28 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2032.codfw.wmnet
* 07:15 marostegui@cumin1002: dbctl commit (dc=all): 'Remove es1022 from dbctl T384566', diff saved to https://phabricator.wikimedia.org/P72239 and previous config saved to /var/cache/conftool/dbconfig/20250123-071529-root.json
* 07:08 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es1021.eqiad.wmnet
* 07:08 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:08 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es1021.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 07:08 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es1021.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 07:01 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 06:55 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts es1021.eqiad.wmnet
* 06:42 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2189 T383709', diff saved to https://phabricator.wikimedia.org/P72237 and previous config saved to /var/cache/conftool/dbconfig/20250123-064241-marostegui.json
* 06:42 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 6:00:00 on db2189.codfw.wmnet with reason: Onsite work
* 06:41 marostegui: Powering off db2189 for onsite maintenance T383709
* 02:35 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 02:10 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1012.eqiad.wmnet with reason: host reimage
* 02:06 andrew@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1012.eqiad.wmnet with reason: host reimage
* 01:50 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 01:49 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 01:27 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1012.eqiad.wmnet with reason: host reimage
* 01:23 andrew@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1012.eqiad.wmnet with reason: host reimage
* 01:06 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 01:06 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 01:00 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 00:59 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 00:54 tzatziki: removing 2 files for legal compliance
* 00:44 tzatziki: removing 1 file for legal complaince
* 00:42 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 00:41 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1012.eqiad.wmnet with OS bullseye
== 2025-01-22 ==
* 23:58 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 23:57 andrew@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 22:24 dmartin@deploy2002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 22:23 dmartin@deploy2002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 22:23 dmartin@deploy2002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 22:22 dmartin@deploy2002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 22:20 dmartin@deploy2002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 22:20 dmartin@deploy2002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 22:20 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1135-1141].eqiad.wmnet
* 22:20 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1135-1141].eqiad.wmnet
* 22:13 dmartin@deploy2002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 22:12 dmartin@deploy2002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 22:12 dmartin@deploy2002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 22:11 dmartin@deploy2002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 22:06 dmartin@deploy2002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 22:06 dmartin@deploy2002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 22:03 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1139.eqiad.wmnet with OS bookworm
* 21:59 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1138.eqiad.wmnet with OS bookworm
* 21:55 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1137.eqiad.wmnet with OS bookworm
* 21:52 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1140.eqiad.wmnet with OS bookworm
* 21:49 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1135.eqiad.wmnet with OS bookworm
* 21:45 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1141.eqiad.wmnet with OS bookworm
* 21:44 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1139.eqiad.wmnet with reason: host reimage
* 21:42 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1136.eqiad.wmnet with OS bookworm
* 21:40 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1138.eqiad.wmnet with reason: host reimage
* 21:37 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1137.eqiad.wmnet with reason: host reimage
* 21:36 dzahn@dns1004: END - running authdns-update
* 21:34 cjming: end of UTC late backport window
* 21:34 dzahn@dns1004: START - running authdns-update
* 21:34 cjming@deploy2002: Finished scap sync-world: Backport for gerrit:1113512Add a few more contextual attributes to web base (T373715) (duration: 11m 41s)
* 21:33 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on netflow7001.magru.wmnet with reason: disabling alerts as I'm running gnmic manually rather than with systemd
* 21:33 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1140.eqiad.wmnet with reason: host reimage
* 21:29 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1135.eqiad.wmnet with reason: host reimage
* 21:27 cjming@deploy2002: cjming: Continuing with sync
* 21:27 cjming@deploy2002: cjming: Backport for gerrit:1113512Add a few more contextual attributes to web base (T373715) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:26 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 21:26 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1141.eqiad.wmnet with reason: host reimage
* 21:22 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1136.eqiad.wmnet with reason: host reimage
* 21:22 cjming@deploy2002: Started scap sync-world: Backport for gerrit:1113512Add a few more contextual attributes to web base (T373715)
* 21:20 cjming@deploy2002: Finished scap sync-world: Backport for gerrit:1113511Enable ExLab test 1 experiment to wikitech (T373715) (duration: 13m 22s)
* 21:20 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1135.eqiad.wmnet with reason: host reimage
* 21:20 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1141.eqiad.wmnet with reason: host reimage
* 21:20 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1139.eqiad.wmnet with reason: host reimage
* 21:19 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1140.eqiad.wmnet with reason: host reimage
* 21:19 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1138.eqiad.wmnet with reason: host reimage
* 21:19 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1137.eqiad.wmnet with reason: host reimage
* 21:19 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1136.eqiad.wmnet with reason: host reimage
* 21:14 cjming@deploy2002: cjming: Continuing with sync
* 21:13 cjming@deploy2002: cjming: Backport for gerrit:1113511Enable ExLab test 1 experiment to wikitech (T373715) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:12 andrew@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 21:07 cjming@deploy2002: Started scap sync-world: Backport for gerrit:1113511Enable ExLab test 1 experiment to wikitech (T373715)
* 21:04 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1135
* 21:04 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1135
* 21:04 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1135.eqiad.wmnet with OS bookworm
* 21:04 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1141
* 21:04 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1141
* 21:04 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1141.eqiad.wmnet with OS bookworm
* 21:04 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1139
* 21:04 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1139
* 21:04 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1139.eqiad.wmnet with OS bookworm
* 21:04 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1140
* 21:03 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1140
* 21:03 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1140.eqiad.wmnet with OS bookworm
* 21:03 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1138
* 21:03 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1138
* 21:03 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1138.eqiad.wmnet with OS bookworm
* 21:03 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1137
* 21:03 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1137
* 21:03 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1137.eqiad.wmnet with OS bookworm
* 21:03 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1136
* 21:03 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1136
* 21:03 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1136.eqiad.wmnet with OS bookworm
* 21:02 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1135.eqiad.wmnet wikikube-worker1136.eqiad.wmnet wikikube-worker1137.eqiad.wmnet wikikube-worker1138.eqiad.wmnet wikikube-worker1139.eqiad.wmnet wikikube-worker1140.eqiad.wmnet wikikube-worker1141.eqiad.wmnet on all recursors
* 21:02 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1135.eqiad.wmnet wikikube-worker1136.eqiad.wmnet wikikube-worker1137.eqiad.wmnet wikikube-worker1138.eqiad.wmnet wikikube-worker1139.eqiad.wmnet wikikube-worker1140.eqiad.wmnet wikikube-worker1141.eqiad.wmnet on all recursors
* 21:02 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1488 to wikikube-worker1141
* 21:01 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1141
* 21:00 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1141
* 21:00 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 21:00 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1488 to wikikube-worker1141 - kamila@cumin1002"
* 21:00 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1488 to wikikube-worker1141 - kamila@cumin1002"
* 20:59 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1137
* 20:58 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1487 to wikikube-worker1140
* 20:58 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1137
* 20:58 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1140
* 20:57 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 20:56 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1140
* 20:56 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1488 to wikikube-worker1141
* 20:56 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 20:56 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1487 to wikikube-worker1140 - kamila@cumin1002"
* 20:56 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1487 to wikikube-worker1140 - kamila@cumin1002"
* 20:53 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1486 to wikikube-worker1139
* 20:53 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1139
* 20:52 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 20:51 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1139
* 20:51 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 20:51 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1486 to wikikube-worker1139 - kamila@cumin1002"
* 20:51 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1487 to wikikube-worker1140
* 20:51 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1486 to wikikube-worker1139 - kamila@cumin1002"
* 20:49 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1485 to wikikube-worker1138
* 20:48 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1138
* 20:48 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 20:47 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1486 to wikikube-worker1139
* 20:47 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1138
* 20:47 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 20:47 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1485 to wikikube-worker1138 - kamila@cumin1002"
* 20:47 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1485 to wikikube-worker1138 - kamila@cumin1002"
* 20:45 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1484 to wikikube-worker1137
* 20:44 kamila@cumin1002: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wikikube-worker1137
* 20:43 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 20:43 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1485 to wikikube-worker1138
* 20:42 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1137
* 20:42 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 20:42 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1484 to wikikube-worker1137 - kamila@cumin1002"
* 20:42 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1484 to wikikube-worker1137 - kamila@cumin1002"
* 20:40 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1483 to wikikube-worker1136
* 20:40 aokoth@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Security Update
* 20:40 dzahn@cumin2002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: security release 
* 20:40 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1136
* 20:39 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 20:38 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1136
* 20:38 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 20:38 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1483 to wikikube-worker1136 - kamila@cumin1002"
* 20:38 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1483 to wikikube-worker1136 - kamila@cumin1002"
* 20:37 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1484 to wikikube-worker1137
* 20:36 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1482 to wikikube-worker1135
* 20:35 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1135
* 20:34 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 20:34 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1135
* 20:34 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 20:34 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1482 to wikikube-worker1135 - kamila@cumin1002"
* 20:34 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1483 to wikikube-worker1136
* 20:34 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1482 to wikikube-worker1135 - kamila@cumin1002"
* 20:32 aokoth@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Update
* 20:31 aokoth@cumin1002: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1003.wikimedia.org with reason: Security Update
* 20:31 aokoth@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Update
* 20:31 aokoth@cumin1002: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1003.wikimedia.org with reason: Security Update
* 20:31 aokoth@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Update
* 20:31 aokoth@cumin1002: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1003.wikimedia.org with reason: Security Update
* 20:31 aokoth@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Update
* 20:30 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 20:30 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1482 to wikikube-worker1135
* 20:12 ebysans@deploy2002: Started deploy [analytics/refinery@28dce47]: Temp accounts deployment [analytics/refinery@28dce471]
* 19:55 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 19:54 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 19:49 ladsgroup@deploy2002: Finished scap sync-world: Backport for gerrit:1113469file migration: Set group0 to write both (T384481) (duration: 12m 41s)
* 19:42 ladsgroup@deploy2002: ladsgroup: Continuing with sync
* 19:39 ladsgroup@deploy2002: ladsgroup: Backport for gerrit:1113469file migration: Set group0 to write both (T384481) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 19:38 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 19:37 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 19:36 ladsgroup@deploy2002: Started scap sync-world: Backport for gerrit:1113469file migration: Set group0 to write both (T384481)
* 19:33 ejegg: payments-wiki upgraded from to 
* 19:17 andrew@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts cloudnet1008-dev.eqiad.wmnet
* 19:17 andrew@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 19:15 andrew@cumin1002: START - Cookbook sre.dns.netbox
* 19:09 andrew@cumin1002: START - Cookbook sre.hosts.decommission for hosts cloudnet1008-dev.eqiad.wmnet
* 19:08 andrew@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts cloudnet1007-dev.eqiad.wmnet
* 19:08 andrew@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 19:05 andrew@cumin1002: START - Cookbook sre.dns.netbox
* 19:03 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 18:59 andrew@cumin1002: START - Cookbook sre.hosts.decommission for hosts cloudnet1007-dev.eqiad.wmnet
* 18:59 andrew@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts cloudnet1008-dev.eqiad.wmnet
* 18:59 andrew@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:57 andrew@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts cloudnet1007-dev.eqiad.wmnet
* 18:57 andrew@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:57 andrew@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet1007-dev.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
* 18:57 andrew@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudnet1007-dev.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
* 18:56 andrew@cumin1002: START - Cookbook sre.dns.netbox
* 18:51 andrew@cumin1002: START - Cookbook sre.dns.netbox
* 18:43 andrew@cumin1002: START - Cookbook sre.hosts.decommission for hosts cloudnet1008-dev.eqiad.wmnet
* 18:43 andrew@cumin1002: START - Cookbook sre.hosts.decommission for hosts cloudnet1007-dev.eqiad.wmnet
* 18:35 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 18:25 swfrench@deploy2002: Finished scap sync-world: Backport for gerrit:1080388Add variables for incremental enrollment in PHP 8.1 (T377042) (duration: 22m 03s)
* 18:22 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 18:22 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 18:16 swfrench@deploy2002: swfrench: Continuing with sync
* 18:16 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 18:16 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 18:15 swfrench-wmf: verified PHP_ENGINE / PHP_ENGINE_STICKY enrollment behavior in mwdebug - T377042
* 18:12 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1129-1134].eqiad.wmnet
* 18:12 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1129-1134].eqiad.wmnet
* 18:08 swfrench@deploy2002: swfrench: Backport for gerrit:1080388Add variables for incremental enrollment in PHP 8.1 (T377042) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 18:03 swfrench@deploy2002: Started scap sync-world: Backport for gerrit:1080388Add variables for incremental enrollment in PHP 8.1 (T377042)
* 18:02 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 18:01 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 17:58 tchin@deploy2002: Finished deploy [airflow-dags/analytics@07104ff]: Deploying latest dags for analytics airflow instance T357684 (duration: 01m 53s)
* 17:58 tchin@deploy2002: Started deploy [airflow-dags/analytics@07104ff]: Deploying latest dags for analytics airflow instance T357684
* 17:56 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1130.eqiad.wmnet with OS bookworm
* 17:55 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 17:55 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 17:35 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1130.eqiad.wmnet with reason: host reimage
* 17:32 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1130.eqiad.wmnet with reason: host reimage
* 17:31 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1134.eqiad.wmnet with OS bookworm
* 17:26 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1132.eqiad.wmnet with OS bookworm
* 17:23 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 17:23 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 17:21 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1133.eqiad.wmnet with OS bookworm
* 17:17 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1131.eqiad.wmnet with OS bookworm
* 17:16 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1130
* 17:16 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1130
* 17:16 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1130.eqiad.wmnet with OS bookworm
* 17:14 kamila@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker1130.eqiad.wmnet with OS bookworm
* 17:14 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1129.eqiad.wmnet with OS bookworm
* 17:12 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1134.eqiad.wmnet with reason: host reimage
* 17:08 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1134.eqiad.wmnet with reason: host reimage
* 17:06 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1132.eqiad.wmnet with reason: host reimage
* 17:02 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1133.eqiad.wmnet with reason: host reimage
* 16:58 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1131.eqiad.wmnet with reason: host reimage
* 16:55 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1129.eqiad.wmnet with reason: host reimage
* 16:53 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1134
* 16:53 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1134
* 16:53 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1134.eqiad.wmnet with OS bookworm
* 16:52 kamila@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker1134.eqiad.wmnet with OS bookworm
* 16:52 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1133.eqiad.wmnet with reason: host reimage
* 16:52 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1132.eqiad.wmnet with reason: host reimage
* 16:52 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1131.eqiad.wmnet with reason: host reimage
* 16:51 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1129.eqiad.wmnet with reason: host reimage
* 16:49 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 16:49 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 16:41 vgutierrez@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:41 vgutierrez@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: clean up test-lb IPs - vgutierrez@cumin1002"
* 16:41 vgutierrez@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: clean up test-lb IPs - vgutierrez@cumin1002"
* 16:37 vgutierrez@cumin1002: START - Cookbook sre.dns.netbox
* 16:36 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1134
* 16:36 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1134
* 16:36 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1134.eqiad.wmnet with OS bookworm
* 16:36 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1133
* 16:36 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1133
* 16:36 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1133.eqiad.wmnet with OS bookworm
* 16:36 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1132
* 16:36 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1132
* 16:36 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1132.eqiad.wmnet with OS bookworm
* 16:36 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1131
* 16:36 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1131
* 16:36 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1131.eqiad.wmnet with OS bookworm
* 16:36 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1130
* 16:36 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1130
* 16:36 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1130.eqiad.wmnet with OS bookworm
* 16:36 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1129
* 16:35 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1129
* 16:35 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1129.eqiad.wmnet with OS bookworm
* 16:33 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1129.eqiad.wmnet wikikube-worker1130.eqiad.wmnet wikikube-worker1131.eqiad.wmnet wikikube-worker1132.eqiad.wmnet wikikube-worker1133.eqiad.wmnet wikikube-worker1134.eqiad.wmnet on all recursors
* 16:33 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1129.eqiad.wmnet wikikube-worker1130.eqiad.wmnet wikikube-worker1131.eqiad.wmnet wikikube-worker1132.eqiad.wmnet wikikube-worker1133.eqiad.wmnet wikikube-worker1134.eqiad.wmnet on all recursors
* 16:33 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1481 to wikikube-worker1134
* 16:32 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1134
* 16:31 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1134
* 16:31 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:31 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1481 to wikikube-worker1134 - kamila@cumin1002"
* 16:30 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1481 to wikikube-worker1134 - kamila@cumin1002"
* 16:28 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1480 to wikikube-worker1133
* 16:28 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1133
* 16:27 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 16:26 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1133
* 16:26 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:26 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1480 to wikikube-worker1133 - kamila@cumin1002"
* 16:26 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1480 to wikikube-worker1133 - kamila@cumin1002"
* 16:26 vgutierrez@cumin1002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-high-traffic1 (T384486)
* 16:24 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1479 to wikikube-worker1132
* 16:23 hnowlan@deploy2002: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 16:23 hnowlan@deploy2002: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 16:23 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1132
* 16:23 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1481 to wikikube-worker1134
* 16:22 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 16:22 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1132
* 16:22 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:22 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1479 to wikikube-worker1132 - kamila@cumin1002"
* 16:22 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1480 to wikikube-worker1133
* 16:22 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1479 to wikikube-worker1132 - kamila@cumin1002"
* 16:20 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 16:20 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-high-traffic1 (T384486)
* 16:20 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 16:19 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1478 to wikikube-worker1131
* 16:19 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1131
* 16:18 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 16:18 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1479 to wikikube-worker1132
* 16:17 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1131
* 16:17 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:17 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1478 to wikikube-worker1131 - kamila@cumin1002"
* 16:17 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1478 to wikikube-worker1131 - kamila@cumin1002"
* 16:15 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1477 to wikikube-worker1130
* 16:14 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1130
* 16:14 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 16:13 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1478 to wikikube-worker1131
* 16:13 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1130
* 16:13 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:13 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1477 to wikikube-worker1130 - kamila@cumin1002"
* 16:13 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1477 to wikikube-worker1130 - kamila@cumin1002"
* 16:10 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1476 to wikikube-worker1129
* 16:10 moritzm: installing rsync regression updates on bullseye
* 16:10 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1129
* 16:09 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 16:09 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1129
* 16:09 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1477 to wikikube-worker1130
* 16:09 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:09 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1476 to wikikube-worker1129 - kamila@cumin1002"
* 16:08 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1476 to wikikube-worker1129 - kamila@cumin1002"
* 16:08 vgutierrez@cumin1002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs[1020].eqiad.wmnet,lvs5006.eqsin.wmnet,lvs3010.esams.wmnet,lvs7003.magru.wmnet,lvs4010.ulsfo.wmnet} and A:lvs (T384486)
* 16:07 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db2166
* 16:07 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 16:07 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host db2166
* 16:05 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 16:04 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1476 to wikikube-worker1129
* 16:04 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[1476-1481].eqiad.wmnet
* 16:03 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs[1020].eqiad.wmnet,lvs5006.eqsin.wmnet,lvs3010.esams.wmnet,lvs7003.magru.wmnet,lvs4010.ulsfo.wmnet} and A:lvs (T384486)
* 16:02 vgutierrez@cumin1002: END (FAIL) - Cookbook sre.loadbalancer.restart-pybal (exit_code=99) rolling-restart of pybal on A:lvs-secondary (T384486)
* 15:55 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[1476-1481].eqiad.wmnet
* 15:54 vgutierrez@cumin1002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-secondary (T384486)
* 15:24 fceratto@cumin1002: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db2189 slowly with 10 steps - Repool host after fixing indexes and performing OS updates
* 15:22 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd2004.codfw.wmnet to plain
* 15:22 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd2004.codfw.wmnet to plain
* 15:21 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2032.codfw.wmnet
* 15:21 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2032.codfw.wmnet
* 15:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd2004.codfw.wmnet to drbd
* 15:11 swfrench-wmf: switched dbctl pc section objects to flavor "parsercache" - T383324
* 14:56 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd2004.codfw.wmnet to drbd
* 14:47 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2032.codfw.wmnet
* 14:45 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2032.codfw.wmnet
* 14:28 urbanecm@deploy2002: Finished scap sync-world: Backport for gerrit:1112219Disable sidebar cache on the auth domain (T383916) (duration: 14m 06s)
* 14:23 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2021.codfw.wmnet to cluster codfw and group B
* 14:22 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2021.codfw.wmnet to cluster codfw and group B
* 14:21 urbanecm@deploy2002: urbanecm, matmarex: Continuing with sync
* 14:20 urbanecm@deploy2002: urbanecm, matmarex: Backport for gerrit:1112219Disable sidebar cache on the auth domain (T383916) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:14 urbanecm@deploy2002: Started scap sync-world: Backport for gerrit:1112219Disable sidebar cache on the auth domain (T383916)
* 14:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2021.codfw.wmnet
* 14:05 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2021.codfw.wmnet
* 14:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2021.codfw.wmnet with OS bookworm
* 13:56 urbanecm: Deployed security patch for T384244
* 13:46 federico3: Removing db2133 from zarcillo T384343
* 13:41 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2021.codfw.wmnet with reason: host reimage
* 13:40 stran@deploy2002: helmfile [codfw] DONE helmfile.d/services/ipoid: apply
* 13:39 stran@deploy2002: helmfile [codfw] START helmfile.d/services/ipoid: apply
* 13:39 stran@deploy2002: helmfile [eqiad] DONE helmfile.d/services/ipoid: apply
* 13:38 stran@deploy2002: helmfile [eqiad] START helmfile.d/services/ipoid: apply
* 13:38 stran@deploy2002: helmfile [staging] DONE helmfile.d/services/ipoid: apply
* 13:38 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2021.codfw.wmnet with reason: host reimage
* 13:38 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2133.codfw.wmnet
* 13:37 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:37 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2133.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 13:37 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2133.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 13:37 stran@deploy2002: helmfile [staging] START helmfile.d/services/ipoid: apply
* 13:34 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 13:29 urbanecm: Deploying security patch for T384244
* 13:29 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2133.codfw.wmnet
* 13:15 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on netflow7001.magru.wmnet with reason: disabling alerts as I'm running gnmic manually rather than with systemd
* 13:08 fceratto@cumin1002: START - Cookbook sre.mysql.pool db2189 slowly with 10 steps - Repool host after fixing indexes and performing OS updates
* 13:08 federico3: repooling db2189 as per T384202
* 12:48 root@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1205.eqiad.wmnet with OS bookworm
* 12:47 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2021.codfw.wmnet with OS bookworm
* 12:34 Amir1: creating new schema of file tables everywhere (T368113)
* 12:25 root@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1205.eqiad.wmnet with reason: host reimage
* 12:24 hnowlan: disabling puppet on A:cp to test r/1113178
* 12:22 root@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1205.eqiad.wmnet with reason: host reimage
* 12:20 mvolz@deploy2002: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
* 12:19 mvolz@deploy2002: helmfile [eqiad] START helmfile.d/services/zotero: apply
* 12:18 mvolz@deploy2002: helmfile [codfw] DONE helmfile.d/services/zotero: apply
* 12:17 mvolz@deploy2002: helmfile [codfw] START helmfile.d/services/zotero: apply
* 12:17 mvolz@deploy2002: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 12:16 mvolz@deploy2002: helmfile [staging] START helmfile.d/services/zotero: apply
* 12:11 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2021.codfw.wmnet with reason: remove from cluster for reimage
* 12:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2021.codfw.wmnet
* 12:05 root@cumin1002: START - Cookbook sre.hosts.reimage for host db1205.eqiad.wmnet with OS bookworm
* 12:02 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1205.eqiad.wmnet with reason: os upgrade
* 11:43 vgutierrez: testing acme-chief 0.38 in acmechief-test1001
* 11:34 marostegui@cumin1002: dbctl commit (dc=all): 'db2175 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72220 and previous config saved to /var/cache/conftool/dbconfig/20250122-113404-root.json
* 11:32 urbanecm@deploy2002: Finished scap sync-world: Backport for gerrit:1113423ValidatorFactory: Allow extensions to register validators (T384246), gerrit:1113424ValidatorFactory: Allow extensions to register validators (T384246) (duration: 11m 55s)
* 11:20 urbanecm@deploy2002: Started scap sync-world: Backport for gerrit:1113423ValidatorFactory: Allow extensions to register validators (T384246), gerrit:1113424ValidatorFactory: Allow extensions to register validators (T384246)
* 11:18 marostegui@cumin1002: dbctl commit (dc=all): 'db2175 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72218 and previous config saved to /var/cache/conftool/dbconfig/20250122-111859-root.json
* 11:14 marostegui@cumin1002: dbctl commit (dc=all): 'Remove es1021 from dbctl T384418', diff saved to https://phabricator.wikimedia.org/P72217 and previous config saved to /var/cache/conftool/dbconfig/20250122-111428-root.json
* 11:10 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2166.codfw.wmnet with reason: Onsite work
* 11:10 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2166 T383709', diff saved to https://phabricator.wikimedia.org/P72216 and previous config saved to /var/cache/conftool/dbconfig/20250122-111019-marostegui.json
* 11:03 marostegui@cumin1002: dbctl commit (dc=all): 'db2175 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72215 and previous config saved to /var/cache/conftool/dbconfig/20250122-110354-root.json
* 10:59 marostegui: Deploy schema change in codfw x1 with replication on the master dbmaint T381759
* 10:48 marostegui@cumin1002: dbctl commit (dc=all): 'db2175 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72214 and previous config saved to /var/cache/conftool/dbconfig/20250122-104848-root.json
* 10:38 topranks: disable-pupept on netflow7001 to run gnmic in foregrand for debug/development T369384
* 10:38 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on netflow7001.magru.wmnet with reason: disabling alerts as I'm running gnmic manually rather than with systemd
* 10:33 marostegui@cumin1002: dbctl commit (dc=all): 'db2175 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72213 and previous config saved to /var/cache/conftool/dbconfig/20250122-103342-root.json
* 10:03 root@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1240.eqiad.wmnet with OS bookworm
* 09:40 root@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1240.eqiad.wmnet with reason: host reimage
* 09:38 root@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1240.eqiad.wmnet with reason: host reimage
* 09:21 root@cumin1002: START - Cookbook sre.hosts.reimage for host db1240.eqiad.wmnet with OS bookworm
* 09:20 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2021.codfw.wmnet
* 09:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2021.codfw.wmnet
* 09:16 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1240.eqiad.wmnet with reason: os upgrade
* 09:15 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2021.codfw.wmnet
* 08:24 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2019.codfw.wmnet to cluster codfw and group B
* 08:22 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2019.codfw.wmnet to cluster codfw and group B
* 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2019.codfw.wmnet
* 08:11 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2019.codfw.wmnet
* 06:51 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1021 T384418', diff saved to https://phabricator.wikimedia.org/P72212 and previous config saved to /var/cache/conftool/dbconfig/20250122-065157-marostegui.json
* 06:20 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2175.codfw.wmnet with reason: rebuilding index
* 02:35 cstone: civicrm upgraded from to 
== 2025-01-21 ==
* 22:06 cjming: end of UTC late backport window
* 22:02 cjming@deploy2002: Finished scap sync-world: Backport for gerrit:1113192Fix schema version for CTR instrument (T384333) (duration: 12m 05s)
* 21:55 cjming@deploy2002: cjming: Continuing with sync
* 21:55 cjming@deploy2002: cjming: Backport for gerrit:1113192Fix schema version for CTR instrument (T384333) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:50 cjming@deploy2002: Started scap sync-world: Backport for gerrit:1113192Fix schema version for CTR instrument (T384333)
* 21:39 cjming@deploy2002: Finished scap sync-world: Backport for gerrit:1112838cawiki: Create templateeditor & protection level (T384145) (duration: 13m 24s)
* 21:38 eileen: civicrm upgraded from to 
* 21:32 cjming@deploy2002: zhaofjx, cjming: Continuing with sync
* 21:32 cjming@deploy2002: zhaofjx, cjming: Backport for gerrit:1112838cawiki: Create templateeditor & protection level (T384145) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:25 cjming@deploy2002: Started scap sync-world: Backport for gerrit:1112838cawiki: Create templateeditor & protection level (T384145)
* 21:23 cjming@deploy2002: Finished scap sync-world: Backport for gerrit:1112101Enable Vector 2022 and dark mode on Azerbaijani wikis (T383942) (duration: 15m 06s)
* 21:16 cjming@deploy2002: cjming, jdlrobson: Continuing with sync
* 21:14 cjming@deploy2002: cjming, jdlrobson: Backport for gerrit:1112101Enable Vector 2022 and dark mode on Azerbaijani wikis (T383942) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:08 cjming@deploy2002: Started scap sync-world: Backport for gerrit:1112101Enable Vector 2022 and dark mode on Azerbaijani wikis (T383942)
* 20:35 ecarg@deploy2002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 20:35 ecarg@deploy2002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 20:14 herron@cumin1002: dbctl commit (dc=all): 'depool db2175', diff saved to https://phabricator.wikimedia.org/P72208 and previous config saved to /var/cache/conftool/dbconfig/20250121-201444-herron.json
* 19:43 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on netflow7001.magru.wmnet with reason: disabling alerts as I'm running gnmic manually rather than with systemd
* 19:32 jebe@deploy2002: Finished deploy [airflow-dags/analytics_product@0aa9d7c]: (no justification provided) (duration: 00m 35s)
* 19:31 jebe@deploy2002: Started deploy [airflow-dags/analytics_product@0aa9d7c]: (no justification provided)
* 19:26 brennen@deploy2002: rebuilt and synchronized wikiversions files: group0 to 1.44.0-wmf.13 refs T382364
* 19:25 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 19:25 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 18:49 swfrench-wmf: finished running puppet on A:cp-text for ATS mapping change - T377042
* 18:38 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1126.eqiad.wmnet with OS bookworm
* 18:35 cmooney@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on netflow7001.magru.wmnet with reason: disabling alerts as I'm running gnmic manually rather than with systemd
* 17:12 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1128.eqiad.wmnet with reason: host reimage
* 17:09 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1125.eqiad.wmnet with reason: host reimage
* 17:06 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1124.eqiad.wmnet with reason: host reimage
* 17:05 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics: apply
* 17:04 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics: apply
* 17:03 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1123.eqiad.wmnet with reason: host reimage
* 17:00 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1128.eqiad.wmnet with reason: host reimage
* 17:00 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1127.eqiad.wmnet with reason: host reimage
* 17:00 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1124.eqiad.wmnet with reason: host reimage
* 17:00 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1125.eqiad.wmnet with reason: host reimage
* 16:59 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1123.eqiad.wmnet with reason: host reimage
* 16:54 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 16:54 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 16:44 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1128
* 16:44 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1128
* 16:44 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1128.eqiad.wmnet with OS bookworm
* 16:44 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1127
* 16:44 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1127
* 16:44 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1127.eqiad.wmnet with OS bookworm
* 16:44 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1126
* 16:44 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1126
* 16:44 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1126.eqiad.wmnet with OS bookworm
* 16:44 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1125
* 16:44 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1125
* 16:44 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1125.eqiad.wmnet with OS bookworm
* 16:44 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1124
* 16:44 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1124
* 16:44 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1124.eqiad.wmnet with OS bookworm
* 16:44 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1123
* 16:44 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1123
* 16:44 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1123.eqiad.wmnet with OS bookworm
* 16:43 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2019.codfw.wmnet with OS bookworm
* 16:35 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1123.eqiad.wmnet wikikube-worker1124.eqiad.wmnet wikikube-worker1125.eqiad.wmnet wikikube-worker1126.eqiad.wmnet wikikube-worker1127.eqiad.wmnet wikikube-worker1128.eqiad.wmnet on all recursors
* 16:35 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1123.eqiad.wmnet wikikube-worker1124.eqiad.wmnet wikikube-worker1125.eqiad.wmnet wikikube-worker1126.eqiad.wmnet wikikube-worker1127.eqiad.wmnet wikikube-worker1128.eqiad.wmnet on all recursors
* 16:34 eevans@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for restbase2037.codfw.wmnet
* 16:34 eevans@cumin1002: START - Cookbook sre.hosts.remove-downtime for restbase2037.codfw.wmnet
* 16:34 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1475 to wikikube-worker1128
* 16:34 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1128
* 16:33 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1128
* 16:33 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:33 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1475 to wikikube-worker1128 - kamila@cumin1002"
* 16:32 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1475 to wikikube-worker1128 - kamila@cumin1002"
* 16:30 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1474 to wikikube-worker1127
* 16:29 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1127
* 16:28 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 16:28 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1127
* 16:28 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:28 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1474 to wikikube-worker1127 - kamila@cumin1002"
* 16:28 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1475 to wikikube-worker1128
* 16:27 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1474 to wikikube-worker1127 - kamila@cumin1002"
* 16:25 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1473 to wikikube-worker1126
* 16:24 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1126
* 16:23 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 16:23 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1126
* 16:23 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:23 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1473 to wikikube-worker1126 - kamila@cumin1002"
* 16:23 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1474 to wikikube-worker1127
* 16:23 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2019.codfw.wmnet with reason: host reimage
* 16:23 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1473 to wikikube-worker1126 - kamila@cumin1002"
* 16:20 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1472 to wikikube-worker1125
* 16:20 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics: apply
* 16:20 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1125
* 16:19 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics: apply
* 16:19 papaul: power down ms-be2088 for maintenance
* 16:19 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2019.codfw.wmnet with reason: host reimage
* 16:19 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 16:19 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1473 to wikikube-worker1126
* 16:19 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1125
* 16:18 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:18 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1472 to wikikube-worker1125 - kamila@cumin1002"
* 16:18 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1472 to wikikube-worker1125 - kamila@cumin1002"
* 16:16 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1471 to wikikube-worker1124
* 16:15 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1124
* 16:15 jynus@dns1004: END - running authdns-update
* 16:14 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 16:14 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1124
* 16:14 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:14 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1471 to wikikube-worker1124 - kamila@cumin1002"
* 16:14 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1472 to wikikube-worker1125
* 16:14 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1471 to wikikube-worker1124 - kamila@cumin1002"
* 16:13 jynus@dns1004: START - running authdns-update
* 16:11 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1470 to wikikube-worker1123
* 16:11 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1123
* 16:10 moritzm: installing gstreamer1.0 security updates
* 16:10 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 16:10 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1123
* 16:10 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:10 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1470 to wikikube-worker1123 - kamila@cumin1002"
* 16:10 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1471 to wikikube-worker1124
* 16:09 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1470 to wikikube-worker1123 - kamila@cumin1002"
* 16:06 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 16:05 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1470 to wikikube-worker1123
* 16:04 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics: apply
* 16:03 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[1470-1475].eqiad.wmnet
* 16:03 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics: apply
* 16:03 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main2010.codfw.wmnet
* 16:03 jayme@cumin1002: START - Cookbook sre.hosts.remove-downtime for kafka-main2010.codfw.wmnet
* 15:55 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[1470-1475].eqiad.wmnet
* 15:52 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host kafka-main2010
* 15:52 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host kafka-main2010
* 15:52 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd2004-dev
* 15:52 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd2004-dev
* 15:47 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2019.codfw.wmnet with OS bookworm
* 15:43 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:41 jhancock@cumin2002: START - Cookbook sre.dns.netbox
* 15:28 Lucas_WMDE: UTC afternoon backport+config window done
* 15:28 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics: apply
* 15:27 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1112261Add known-good regexes for WikibaseQualityConstraints (T380751) (duration: 29m 44s)
* 15:27 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics: apply
* 15:25 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics: apply
* 15:24 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics: apply
* 15:20 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, audreypenven: Continuing with sync
* 15:18 jayme@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-main2010.codfw.wmnet with reason: Server moving within rack
* 15:07 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics: apply
* 15:07 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics: apply
* 15:04 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 15:04 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, audreypenven: Backport for gerrit:1112261Add known-good regexes for WikibaseQualityConstraints (T380751) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 15:04 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 14:58 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1112261Add known-good regexes for WikibaseQualityConstraints (T380751)
* 14:54 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1107936enable 2 factor authentication for enwiki page movers (T382879) (duration: 16m 10s)
* 14:49 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 14:49 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 14:47 lucaswerkmeister-wmde@deploy2002: novemlinguae, lucaswerkmeister-wmde: Continuing with sync
* 14:45 lucaswerkmeister-wmde@deploy2002: novemlinguae, lucaswerkmeister-wmde: Backport for gerrit:1107936enable 2 factor authentication for enwiki page movers (T382879) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:44 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2019.codfw.wmnet
* 14:41 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2019.codfw.wmnet with reason: remove from cluster for reimage
* 14:38 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1107936enable 2 factor authentication for enwiki page movers (T382879)
* 14:36 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1111932Remove KartographerParsoidSupport flag from configuration (T340134) (duration: 19m 46s)
* 14:35 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 14:35 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 14:27 lucaswerkmeister-wmde@deploy2002: ihurbain, lucaswerkmeister-wmde: Continuing with sync
* 14:26 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics: apply
* 14:25 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics: apply
* 14:24 lucaswerkmeister-wmde@deploy2002: ihurbain, lucaswerkmeister-wmde: Backport for gerrit:1111932Remove KartographerParsoidSupport flag from configuration (T340134) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:23 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics: apply
* 14:22 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics: apply
* 14:16 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1111932Remove KartographerParsoidSupport flag from configuration (T340134)
* 14:10 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
* 14:09 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
* 14:09 btullis@cumin1002: END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) for Druid analytics cluster: Roll restart of Druid jvm daemons.
* 13:55 Emperor: hard-reboot ms-fe1014
* 13:54 mvernon@cumin2002: conftool action : set/pooled=no; selector: name=ms-fe1014.eqiad.wmnet
* 13:43 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics: apply
* 13:43 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics: apply
* 13:27 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics: apply
* 13:26 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics: apply
* 13:22 btullis@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-launcher1002.eqiad.wmnet with reason: Migrating to kubernetes
* 13:22 btullis@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-launcher1002.eqiad.wmnet with reason: Migrating to kubernetes
* 13:20 btullis@cumin1002: START - Cookbook sre.druid.roll-restart-workers for Druid analytics cluster: Roll restart of Druid jvm daemons.
* 13:17 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
* 13:16 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
* 13:01 hnowlan@deploy2002: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 13:01 hnowlan@deploy2002: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 12:59 hnowlan@deploy2002: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 12:59 hnowlan@deploy2002: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 12:48 marostegui@cumin1002: dbctl commit (dc=all): 'db2203 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72198 and previous config saved to /var/cache/conftool/dbconfig/20250121-124857-root.json
* 12:47 marostegui@cumin1002: dbctl commit (dc=all): 'db2216 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72197 and previous config saved to /var/cache/conftool/dbconfig/20250121-124750-root.json
* 12:38 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2019.codfw.wmnet
* 12:33 marostegui@cumin1002: dbctl commit (dc=all): 'db2203 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72194 and previous config saved to /var/cache/conftool/dbconfig/20250121-123352-root.json
* 12:33 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd2004.codfw.wmnet to plain
* 12:32 marostegui@cumin1002: dbctl commit (dc=all): 'db2216 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72193 and previous config saved to /var/cache/conftool/dbconfig/20250121-123245-root.json
* 12:32 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd2004.codfw.wmnet to plain
* 12:32 hnowlan@deploy2002: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 12:32 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2019.codfw.wmnet
* 12:32 hnowlan@deploy2002: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 12:31 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2019.codfw.wmnet
* 12:29 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd2004.codfw.wmnet to drbd
* 12:27 hnowlan@deploy2002: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:27 hnowlan@deploy2002: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:18 marostegui@cumin1002: dbctl commit (dc=all): 'db2203 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72190 and previous config saved to /var/cache/conftool/dbconfig/20250121-121847-root.json
* 12:17 marostegui@cumin1002: dbctl commit (dc=all): 'db2216 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72189 and previous config saved to /var/cache/conftool/dbconfig/20250121-121739-root.json
* 12:15 kart_: Updated cxserver to 2025-01-20-172318-production (T377966, T377813)
* 12:15 kartik@deploy2002: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
* 12:14 kartik@deploy2002: helmfile [eqiad] START helmfile.d/services/cxserver: apply
* 12:10 kartik@deploy2002: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
* 12:09 kartik@deploy2002: helmfile [codfw] START helmfile.d/services/cxserver: apply
* 12:09 fceratto@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2189.codfw.wmnet
* 12:08 hnowlan@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on restbase2037.codfw.wmnet with reason: Memory issues, rebooting frequently. Depooled. T383820
* 12:05 federico3: updating db2189.codfw.wmnet for https://phabricator.wikimedia.org/T384202
* 12:05 kartik@deploy2002: helmfile [staging] DONE helmfile.d/services/cxserver: apply
* 12:04 fceratto@cumin1002: START - Cookbook sre.mysql.upgrade for db2189.codfw.wmnet
* 12:03 marostegui@cumin1002: dbctl commit (dc=all): 'db2203 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72187 and previous config saved to /var/cache/conftool/dbconfig/20250121-120341-root.json
* 12:03 kartik@deploy2002: helmfile [staging] START helmfile.d/services/cxserver: apply
* 12:02 marostegui@cumin1002: dbctl commit (dc=all): 'db2216 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72186 and previous config saved to /var/cache/conftool/dbconfig/20250121-120234-root.json
* 12:01 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd2004.codfw.wmnet to drbd
* 11:59 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2019.codfw.wmnet
* 11:59 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2019.codfw.wmnet
* 11:57 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2019.codfw.wmnet
* 11:54 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2019.codfw.wmnet
* 11:48 marostegui@cumin1002: dbctl commit (dc=all): 'db2203 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72185 and previous config saved to /var/cache/conftool/dbconfig/20250121-114836-root.json
* 11:48 hnowlan@cumin2002: conftool action : set/pooled=no; selector: name=restbase2037.codfw.wmnet
* 11:47 marostegui@cumin1002: dbctl commit (dc=all): 'db2216 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72184 and previous config saved to /var/cache/conftool/dbconfig/20250121-114728-root.json
* 11:44 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2189.codfw.wmnet with reason: rebuilding index
* 11:34 jiji@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 11:32 jiji@deploy2002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 11:32 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 11:31 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 11:30 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 11:30 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 11:29 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 11:29 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 11:29 jiji@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 11:28 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 11:28 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 11:27 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 11:26 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 11:26 jiji@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 11:26 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 11:25 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 11:19 jiji@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
* 11:18 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply
* 11:18 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply
* 11:14 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 11:14 jiji@deploy2002: helmfile [codfw] START helmfile.d/services/mw-web: apply
* 11:13 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 11:13 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 11:13 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 11:12 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 11:11 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 11:11 jiji@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
* 11:10 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 11:10 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 11:09 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 11:09 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 11:08 jiji@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 11:01 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2024.codfw.wmnet to cluster codfw and group A
* 11:00 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2024.codfw.wmnet to cluster codfw and group A
* 10:46 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72182 and previous config saved to /var/cache/conftool/dbconfig/20250121-104658-root.json
* 10:45 marostegui@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72181 and previous config saved to /var/cache/conftool/dbconfig/20250121-104559-root.json
* 10:40 topranks: de-pref Chicago routes learnt on on core routers in Dallas
* 10:31 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72180 and previous config saved to /var/cache/conftool/dbconfig/20250121-103153-root.json
* 10:30 marostegui@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72179 and previous config saved to /var/cache/conftool/dbconfig/20250121-103053-root.json
* 10:29 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2024.codfw.wmnet
* 10:26 topranks: adjust VRRP priorities for public and analytics vlans on eqiad CRs to balance traffic
* 10:20 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2024.codfw.wmnet
* 10:16 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72177 and previous config saved to /var/cache/conftool/dbconfig/20250121-101648-root.json
* 10:15 marostegui@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72176 and previous config saved to /var/cache/conftool/dbconfig/20250121-101548-root.json
* 10:11 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics: apply
* 10:11 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics: apply
* 10:03 moritzm: installing python-tornado security updates
* 10:03 moritzm: installing intel-microcode security updates
* 10:01 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72175 and previous config saved to /var/cache/conftool/dbconfig/20250121-100142-root.json
* 10:00 marostegui@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72174 and previous config saved to /var/cache/conftool/dbconfig/20250121-100042-root.json
* 10:00 godog: set udp_localhost-info retention.bytes=300000000000 on kafka-logging (back to original value) - T384233
* 09:58 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2024.codfw.wmnet with OS bookworm
* 09:52 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 09:52 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 09:47 godog: set udp_localhost-info retention.bytes=100000000000 on kafka-logging - T384233
* 09:46 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72173 and previous config saved to /var/cache/conftool/dbconfig/20250121-094637-root.json
* 09:45 marostegui@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72172 and previous config saved to /var/cache/conftool/dbconfig/20250121-094537-root.json
* 09:39 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 09:39 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 09:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2024.codfw.wmnet with reason: host reimage
* 09:33 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2024.codfw.wmnet with reason: host reimage
* 09:32 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 09:32 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 09:30 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 09:30 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.resource-report (exit_code=0)
* 09:30 jmm@cumin2002: START - Cookbook sre.ganeti.resource-report
* 09:30 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 09:30 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 09:29 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 09:29 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 09:10 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2024.codfw.wmnet with OS bookworm
* 09:04 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 09:03 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 08:59 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 08:58 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 08:51 marostegui@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2216.codfw.wmnet onto db2203.codfw.wmnet
* 08:36 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2024.codfw.wmnet with reason: remove from cluster for reimage
* 08:02 kevinbazira@deploy2002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' .
* 07:59 kevinbazira@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' .
* 07:56 kevinbazira@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' .
* 07:42 aqu@deploy2002: Finished deploy [airflow-dags/analytics@868de0c]: 202412 Backfill: Fixes on ExternalTaskMarker experiment (duration: 00m 32s)
* 07:41 aqu@deploy2002: Started deploy [airflow-dags/analytics@868de0c]: 202412 Backfill: Fixes on ExternalTaskMarker experiment
* 07:29 oblivian@cumin1002: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Bugfixes - oblivian@cumin1002"
* 07:29 oblivian@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes - oblivian@cumin1002
* 07:28 oblivian@cumin1002: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes - oblivian@cumin1002
* 07:28 oblivian@cumin1002: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Bugfixes - oblivian@cumin1002"
* 07:02 marostegui@cumin1002: START - Cookbook sre.mysql.clone of db2216.codfw.wmnet onto db2203.codfw.wmnet
* 06:58 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2216.codfw.wmnet with reason: rebuilding index
* 06:56 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2216 T384273', diff saved to https://phabricator.wikimedia.org/P72169 and previous config saved to /var/cache/conftool/dbconfig/20250121-065640-marostegui.json
* 06:52 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2203.codfw.wmnet with reason: rebuilding index
* 06:51 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2203 T383690', diff saved to https://phabricator.wikimedia.org/P72168 and previous config saved to /var/cache/conftool/dbconfig/20250121-065114-marostegui.json
* 06:51 marostegui@dns1006: END - running authdns-update
* 06:49 marostegui@dns1006: START - running authdns-update
* 06:49 marostegui@dns1006: START - running authdns-update
* 05:04 mwpresync@deploy2002: Pruned MediaWiki: 1.44.0-wmf.8 (duration: 04m 55s)
* 05:01 mwpresync@deploy2002: Finished scap sync-world: testwikis to 1.44.0-wmf.13 refs T382364 (duration: 58m 53s)
* 04:02 mwpresync@deploy2002: Started scap sync-world: testwikis to 1.44.0-wmf.13 refs T382364
== 2025-01-20 ==
* 23:37 jynus: run OPTIMIZE TABLE recentchanges on db2207
* 22:14 urbanecm: [urbanecm@deploy2002 ~]$ mwscript-k8s -f namespaceDupes.php -- --wiki=uzwiktionary --fix --add-prefix=BROKEN # T362620, logs posted to the task
* 22:08 urbanecm: Run mwscript-k8s -f namespaceDupes.php -- --wiki=uzwiktionary --fix # T362620 # logs are at P72163
* 22:03 urbanecm@deploy2002: Finished scap sync-world: Backport for gerrit:1079619Update uzwiktionary project namespace and site name (T362620) (duration: 12m 45s)
* 21:56 urbanecm@deploy2002: nmw03, urbanecm: Continuing with sync
* 21:54 urbanecm@deploy2002: nmw03, urbanecm: Backport for gerrit:1079619Update uzwiktionary project namespace and site name (T362620) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:50 urbanecm@deploy2002: Started scap sync-world: Backport for gerrit:1079619Update uzwiktionary project namespace and site name (T362620)
* 21:43 urbanecm@deploy2002: Finished scap sync-world: Backport for gerrit:1110866Add simplewiki to mobile-anon-talk (T383161), gerrit:1112795beta: Enable $wgCampaignEventsEnableEventTopics (T380817) (duration: 13m 21s)
* 21:36 urbanecm@deploy2002: pppery, daimona, urbanecm: Continuing with sync
* 21:35 urbanecm@deploy2002: pppery, daimona, urbanecm: Backport for gerrit:1110866Add simplewiki to mobile-anon-talk (T383161), gerrit:1112795beta: Enable $wgCampaignEventsEnableEventTopics (T380817) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:30 urbanecm@deploy2002: Started scap sync-world: Backport for gerrit:1110866Add simplewiki to mobile-anon-talk (T383161), gerrit:1112795beta: Enable $wgCampaignEventsEnableEventTopics (T380817)
* 19:45 Amir1: dropping blobs table where it's empty (T376627)
* 18:00 elukey@deploy2002: helmfile [staging] DONE helmfile.d/services/kartotherian: sync
* 17:59 elukey@deploy2002: helmfile [staging] START helmfile.d/services/kartotherian: sync
* 17:22 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2024.codfw.wmnet
* 17:02 jdrewniak@deploy2002: Synchronized portals: Wikimedia Portals Update: gerrit:1112805 Bumping portals to master (T128546) (duration: 02m 42s)
* 16:59 jdrewniak@deploy2002: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: gerrit:1112805 Bumping portals to master (T128546) (duration: 13m 40s)
* 16:21 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1117-1122].eqiad.wmnet
* 16:20 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1117-1122].eqiad.wmnet
* 16:05 brouberol@deploy2002: Finished deploy [airflow-dags/search@8c96899]: (no justification provided) (duration: 00m 31s)
* 16:04 brouberol@deploy2002: Started deploy [airflow-dags/search@8c96899]: (no justification provided)
* 16:03 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 16:03 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 16:02 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply
* 16:02 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply
* 16:02 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
* 16:01 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
* 16:01 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 16:01 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 15:53 dcausse@deploy2002: helmfile [eqiad] DONE helmfile.d/services/rdf-streaming-updater: apply
* 15:53 dcausse@deploy2002: helmfile [eqiad] START helmfile.d/services/rdf-streaming-updater: apply
* 15:50 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:50 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:45 moritzm: installing python-tornado security updates
* 15:43 brouberol@deploy2002: Finished deploy [airflow-dags/analytics_test@516e8f2]: (no justification provided) (duration: 00m 21s)
* 15:43 dcausse@deploy2002: helmfile [codfw] DONE helmfile.d/services/rdf-streaming-updater: apply
* 15:43 brouberol@deploy2002: Started deploy [airflow-dags/analytics_test@516e8f2]: (no justification provided)
* 15:43 dcausse@deploy2002: helmfile [codfw] START helmfile.d/services/rdf-streaming-updater: apply
* 15:39 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1112771Make known-good regex check strict (T380751) (duration: 10m 35s)
* 15:37 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw2282.codfw.wmnet
* 15:37 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:37 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw2282.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jelto@cumin1002"
* 15:36 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw2282.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jelto@cumin1002"
* 15:32 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde: Continuing with sync
* 15:32 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde: Backport for gerrit:1112771Make known-good regex check strict (T380751) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 15:29 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1112771Make known-good regex check strict (T380751)
* 15:26 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 15:22 dcausse@deploy2002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
* 15:21 jelto@cumin1002: START - Cookbook sre.hosts.decommission for hosts mw2282.codfw.wmnet
* 15:21 dcausse@deploy2002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
* 15:18 jelto@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host mw2282.codfw.wmnet
* 15:18 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw2282.codfw.wmnet
* 15:18 jelto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on mw2282.codfw.wmnet with reason: decommissioning host
* 15:18 volans: issues power off via mgmt UI for db2131 (failed to power off during decommissioning)
* 15:14 marostegui@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72162 and previous config saved to /var/cache/conftool/dbconfig/20250120-151402-root.json
* 15:04 Lucas_WMDE: UTC afternoon backport+config window done
* 15:02 taavi@deploy2002: Finished scap sync-world: Backport for gerrit:1112333wikitech: Drop obsolete oauthadmin group (T384122), gerrit:1112335wikitech: Drop oathauth group (T384123) (duration: 12m 21s)
* 14:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72161 and previous config saved to /var/cache/conftool/dbconfig/20250120-145856-root.json
* 14:56 taavi@deploy2002: taavi: Continuing with sync
* 14:55 taavi@deploy2002: taavi: Backport for gerrit:1112333wikitech: Drop obsolete oauthadmin group (T384122), gerrit:1112335wikitech: Drop oathauth group (T384123) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:54 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2024.codfw.wmnet
* 14:53 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2024.codfw.wmnet
* 14:51 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2024.codfw.wmnet
* 14:50 taavi@deploy2002: Started scap sync-world: Backport for gerrit:1112333wikitech: Drop obsolete oauthadmin group (T384122), gerrit:1112335wikitech: Drop oathauth group (T384123)
* 14:48 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1112707Add dedicated experimentation lab test module (T373715) (duration: 12m 12s)
* 14:45 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1120.eqiad.wmnet with OS bookworm
* 14:43 marostegui@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72160 and previous config saved to /var/cache/conftool/dbconfig/20250120-144351-root.json
* 14:42 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1119.eqiad.wmnet with OS bookworm
* 14:41 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, cjming: Continuing with sync
* 14:40 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, cjming: Backport for gerrit:1112707Add dedicated experimentation lab test module (T373715) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:40 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1122.eqiad.wmnet with OS bookworm
* 14:38 vgutierrez@dns1004: END - running authdns-update
* 14:36 vgutierrez@dns1004: START - running authdns-update
* 14:36 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1112707Add dedicated experimentation lab test module (T373715)
* 14:33 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1118.eqiad.wmnet with OS bookworm
* 14:30 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1121.eqiad.wmnet with OS bookworm
* 14:30 oblivian@deploy2002: Finished scap sync-world: Backport for gerrit:1109109Use a bespoke database configuration for dumps (T382947) (duration: 18m 47s)
* 14:29 volans@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:05:00 on sretest1002.eqiad.wmnet with reason: testing cumin
* 14:28 marostegui@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72159 and previous config saved to /var/cache/conftool/dbconfig/20250120-142846-root.json
* 14:28 volans: upgraded cumin to v5.0.0 on cumin2002
* 14:27 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1117.eqiad.wmnet with OS bookworm
* 14:27 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1120.eqiad.wmnet with reason: host reimage
* 14:23 oblivian@deploy2002: oblivian: Continuing with sync
* 14:22 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2023.codfw.wmnet to cluster codfw and group A
* 14:22 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1119.eqiad.wmnet with reason: host reimage
* 14:22 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2023.codfw.wmnet to cluster codfw and group A
* 14:20 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2023.codfw.wmnet
* 14:18 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1122.eqiad.wmnet with reason: host reimage
* 14:16 oblivian@deploy2002: oblivian: Backport for gerrit:1109109Use a bespoke database configuration for dumps (T382947) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:14 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1118.eqiad.wmnet with reason: host reimage
* 14:13 marostegui@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72158 and previous config saved to /var/cache/conftool/dbconfig/20250120-141340-root.json
* 14:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2023.codfw.wmnet
* 14:11 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1121.eqiad.wmnet with reason: host reimage
* 14:11 oblivian@deploy2002: Started scap sync-world: Backport for gerrit:1109109Use a bespoke database configuration for dumps (T382947)
* 14:09 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1117.eqiad.wmnet with reason: host reimage
* 14:07 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2023.codfw.wmnet with OS bookworm
* 14:06 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1120.eqiad.wmnet with reason: host reimage
* 14:06 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1121.eqiad.wmnet with reason: host reimage
* 14:06 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1119.eqiad.wmnet with reason: host reimage
* 14:06 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1122.eqiad.wmnet with reason: host reimage
* 14:06 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1118.eqiad.wmnet with reason: host reimage
* 14:06 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1117.eqiad.wmnet with reason: host reimage
* 14:05 kevinbazira@deploy2002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
* 14:02 kevinbazira@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
* 13:58 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply
* 13:57 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply
* 13:57 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 13:56 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 13:56 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
* 13:55 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
* 13:55 kevinbazira@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
* 13:55 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
* 13:54 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
* 13:53 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 13:52 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 13:50 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1122
* 13:50 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1122
* 13:50 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1122.eqiad.wmnet with OS bookworm
* 13:50 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1121
* 13:50 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1121
* 13:50 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1121.eqiad.wmnet with OS bookworm
* 13:50 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1120
* 13:50 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1120
* 13:50 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1120.eqiad.wmnet with OS bookworm
* 13:50 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1119
* 13:50 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1119
* 13:50 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1119.eqiad.wmnet with OS bookworm
* 13:50 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1118
* 13:50 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1118
* 13:50 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1118.eqiad.wmnet with OS bookworm
* 13:50 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1117
* 13:50 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1117
* 13:49 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1117.eqiad.wmnet with OS bookworm
* 13:47 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1117.eqiad.wmnet wikikube-worker1118.eqiad.wmnet wikikube-worker1119.eqiad.wmnet wikikube-worker1120.eqiad.wmnet wikikube-worker1121.eqiad.wmnet wikikube-worker1122.eqiad.wmnet on all recursors
* 13:47 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1117.eqiad.wmnet wikikube-worker1118.eqiad.wmnet wikikube-worker1119.eqiad.wmnet wikikube-worker1120.eqiad.wmnet wikikube-worker1121.eqiad.wmnet wikikube-worker1122.eqiad.wmnet on all recursors
* 13:47 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1469 to wikikube-worker1122
* 13:47 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1122
* 13:46 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2023.codfw.wmnet with reason: host reimage
* 13:44 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1122
* 13:44 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:44 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1469 to wikikube-worker1122 - kamila@cumin1002"
* 13:44 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1469 to wikikube-worker1122 - kamila@cumin1002"
* 13:43 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2023.codfw.wmnet with reason: host reimage
* 13:43 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 13:42 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 13:42 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1468 to wikikube-worker1121
* 13:41 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1121
* 13:40 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 13:40 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1121
* 13:40 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:40 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1468 to wikikube-worker1121 - kamila@cumin1002"
* 13:40 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1469 to wikikube-worker1122
* 13:39 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1468 to wikikube-worker1121 - kamila@cumin1002"
* 13:38 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1467 to wikikube-worker1120
* 13:37 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1120
* 13:36 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 13:36 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1120
* 13:36 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:36 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1467 to wikikube-worker1120 - kamila@cumin1002"
* 13:36 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1468 to wikikube-worker1121
* 13:35 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1467 to wikikube-worker1120 - kamila@cumin1002"
* 13:32 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1466 to wikikube-worker1119
* 13:31 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1119
* 13:31 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 13:30 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1467 to wikikube-worker1120
* 13:30 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1119
* 13:30 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:30 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1466 to wikikube-worker1119 - kamila@cumin1002"
* 13:30 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1466 to wikikube-worker1119 - kamila@cumin1002"
* 13:28 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1465 to wikikube-worker1118
* 13:27 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1118
* 13:26 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1118
* 13:26 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 13:26 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:26 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1465 to wikikube-worker1118 - kamila@cumin1002"
* 13:25 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1465 to wikikube-worker1118 - kamila@cumin1002"
* 13:25 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1466 to wikikube-worker1119
* 13:23 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1464 to wikikube-worker1117
* 13:22 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1117
* 13:21 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 13:21 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1117
* 13:21 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:21 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1464 to wikikube-worker1117 - kamila@cumin1002"
* 13:21 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1465 to wikikube-worker1118
* 13:19 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1464 to wikikube-worker1117 - kamila@cumin1002"
* 13:16 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 13:16 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 13:15 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 13:15 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1464 to wikikube-worker1117
* 13:14 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: rebuilding index
* 13:13 urbanecm@deploy2002: Finished scap sync-world: Backport for [[gerrit:1112207[Growth] Remove tybanner campaigns (T380405)]], [[gerrit:1112208[Growth] Add fundraising- as a prefix for fundraising campaign (T380405)]] (duration: 12m 52s)
* 13:12 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2189.codfw.wmnet with reason: rebuilding
* 13:11 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2189', diff saved to https://phabricator.wikimedia.org/P72157 and previous config saved to /var/cache/conftool/dbconfig/20250120-131157-marostegui.json
* 13:11 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[1464-1469].eqiad.wmnet
* 13:07 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[1464-1469].eqiad.wmnet
* 13:06 urbanecm@deploy2002: urbanecm: Continuing with sync
* 13:05 urbanecm@deploy2002: urbanecm: Backport for [[gerrit:1112207[Growth] Remove tybanner campaigns (T380405)]], [[gerrit:1112208[Growth] Add fundraising- as a prefix for fundraising campaign (T380405)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 13:00 urbanecm@deploy2002: Started scap sync-world: Backport for [[gerrit:1112207[Growth] Remove tybanner campaigns (T380405)]], [[gerrit:1112208[Growth] Add fundraising- as a prefix for fundraising campaign (T380405)]]
* 12:58 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 12:57 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 12:55 federico3: Removing db2131 from zarcillo T384001
* 12:49 fceratto@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts db2131.codfw.wmnet
* 12:49 fceratto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:49 fceratto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2131.codfw.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1002"
* 12:48 fceratto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2131.codfw.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1002"
* 12:47 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2023.codfw.wmnet with OS bookworm
* 12:39 fceratto@cumin1002: START - Cookbook sre.dns.netbox
* 12:37 marostegui@cumin1002: dbctl commit (dc=all): 'Deploy pc7 in both eqiad and codfw T383235', diff saved to https://phabricator.wikimedia.org/P72155 and previous config saved to /var/cache/conftool/dbconfig/20250120-123741-marostegui.json
* 12:34 fceratto@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2131.codfw.wmnet
* 12:13 fceratto@cumin1002: dbctl commit (dc=all): 'Remove db2131 from dbctl T384001', diff saved to https://phabricator.wikimedia.org/P72154 and previous config saved to /var/cache/conftool/dbconfig/20250120-121318-fceratto.json
* 11:48 fceratto@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2131.codfw.wmnet with reason: Downtime db2131
* 11:46 aqu@deploy2002: Finished deploy [airflow-dags/analytics@663a2f0]: 202412 Backfill: Fix ExternalTaskSensor missing filters (duration: 00m 35s)
* 11:45 aqu@deploy2002: Started deploy [airflow-dags/analytics@663a2f0]: 202412 Backfill: Fix ExternalTaskSensor missing filters
* 11:43 zabe@deploy2002: Finished scap sync-world: Backport for gerrit:1112720Fix logo issues (duration: 12m 25s)
* 11:43 fceratto@cumin1002: dbctl commit (dc=all): 'Depool db2131 T384001', diff saved to https://phabricator.wikimedia.org/P72153 and previous config saved to /var/cache/conftool/dbconfig/20250120-114306-fceratto.json
* 11:41 federico3: depooling db2131 as per T384001
* 11:36 zabe@deploy2002: zabe: Continuing with sync
* 11:36 zabe@deploy2002: zabe: Backport for gerrit:1112720Fix logo issues synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 11:31 zabe@deploy2002: Started scap sync-world: Backport for gerrit:1112720Fix logo issues
* 11:23 marostegui@dns1006: END - running authdns-update
* 11:22 marostegui@dns1006: START - running authdns-update
* 11:16 root@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1245.eqiad.wmnet with OS bookworm
* 11:14 jmm@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2023.codfw.wmnet with reason: remove from cluster for reimage
* 11:10 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2023.codfw.wmnet
* 10:54 root@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1245.eqiad.wmnet with reason: host reimage
* 10:50 root@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1245.eqiad.wmnet with reason: host reimage
* 10:34 root@cumin1002: START - Cookbook sre.hosts.reimage for host db1245.eqiad.wmnet with OS bookworm
* 10:27 jynus@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1245.eqiad.wmnet with reason: os upgrade
* 09:51 moritzm: installing intel-microcode security updates
* 09:31 dreamyjazz@deploy2002: Finished scap sync-world: Backport for gerrit:1112220Revert^2 "Pin wgCheckUserEnableTempAccountsOnboardingDialog as false" (duration: 11m 45s)
* 09:24 dreamyjazz@deploy2002: dreamyjazz: Continuing with sync
* 09:24 dreamyjazz@deploy2002: dreamyjazz: Backport for gerrit:1112220Revert^2 "Pin wgCheckUserEnableTempAccountsOnboardingDialog as false" synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 09:22 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster2005.codfw.wmnet to plain
* 09:21 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster2005.codfw.wmnet to plain
* 09:21 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2023.codfw.wmnet
* 09:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2023.codfw.wmnet
* 09:20 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2023.codfw.wmnet
* 09:19 dreamyjazz@deploy2002: Started scap sync-world: Backport for gerrit:1112220Revert^2 "Pin wgCheckUserEnableTempAccountsOnboardingDialog as false"
* 09:19 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster2005.codfw.wmnet to drbd
* 09:16 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2212.codfw.wmnet with reason: rebuilding
* 09:15 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2212', diff saved to https://phabricator.wikimedia.org/P72151 and previous config saved to /var/cache/conftool/dbconfig/20250120-091545-marostegui.json
* 09:14 cjming: end of UTC morning backport window
* 09:13 cjming@deploy2002: Finished scap sync-world: Backport for gerrit:1112105Enable the text experiment on testwiki only (T373715) (duration: 14m 32s)
* 09:05 cjming@deploy2002: cjming: Continuing with sync
* 09:03 jmm@cumin2002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster2005.codfw.wmnet to drbd
* 09:03 cjming@deploy2002: cjming: Backport for gerrit:1112105Enable the text experiment on testwiki only (T373715) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 08:58 cjming@deploy2002: Started scap sync-world: Backport for gerrit:1112105Enable the text experiment on testwiki only (T373715)
* 08:58 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2023.codfw.wmnet
* 08:56 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2023.codfw.wmnet
* 08:55 awight@deploy2002: Finished scap sync-world: Backport for gerrit:1111215Switch to explicit numbering for Parsoid footnote markers (T382310) (duration: 23m 38s)
* 08:45 awight@deploy2002: awight: Continuing with sync
* 08:44 awight@deploy2002: awight: Backport for gerrit:1111215Switch to explicit numbering for Parsoid footnote markers (T382310) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 08:34 moritzm: installing Linux 6.1.124 on Bookworm hosts
* 08:31 awight@deploy2002: Started scap sync-world: Backport for gerrit:1111215Switch to explicit numbering for Parsoid footnote markers (T382310)
* 08:30 moritzm: installing python-aiohttp security updates
* 08:15 marostegui: Deploy schema change on x1 codfw master db2196 with replication dbmaint T384176
* 08:04 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2027.codfw.wmnet to cluster codfw and group A
* 08:03 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2027.codfw.wmnet to cluster codfw and group A
* 07:51 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2027.codfw.wmnet
* 07:43 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti2027.codfw.wmnet
* 07:36 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on pc2017.codfw.wmnet,pc1017.eqiad.wmnet with reason: reorganizing pc7
== 2025-01-18 ==
* 07:40 Ammar: T384109 Ran mwscript-k8s --comment="T384109" -f -- extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=hywiktionary --logwiki=metawiki 'Մարիամ11' 'GrigorYAN'
* 07:40 Ammar: T384109 Ran mwscript-k8s --comment="T384109" -f -- extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=loginwiki --logwiki=metawiki 'Ruhaina Sheikh' 'Lee Ailiseu'
* 01:07 mstyles@deploy2002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:07 mstyles@deploy2002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:07 mstyles@deploy2002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:07 mstyles@deploy2002: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:06 mstyles@deploy2002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:06 mstyles@deploy2002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:05 mstyles@deploy2002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 01:04 mstyles@deploy2002: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 01:02 mstyles@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 01:01 mstyles@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
== 2025-01-17 ==
* 18:18 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:18 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns names for newly assigned wmcs private ipv6 entries - cmooney@cumin1002"
* 18:18 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns names for newly assigned wmcs private ipv6 entries - cmooney@cumin1002"
* 18:13 cmooney@cumin1002: START - Cookbook sre.dns.netbox
* 18:12 cmooney@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 18:08 cmooney@cumin1002: START - Cookbook sre.dns.netbox
* 18:05 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:05 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns names for newly assigned wmcs private ipv6 entries - cmooney@cumin1002"
* 18:05 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns names for newly assigned wmcs private ipv6 entries - cmooney@cumin1002"
* 17:59 cmooney@cumin1002: START - Cookbook sre.dns.netbox
* 17:44 hnowlan@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw[2259,2263-2266].codfw.wmnet
* 17:44 hnowlan@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:44 hnowlan@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[2259,2263-2266].codfw.wmnet decommissioned, removing all IPs except the asset tag one - hnowlan@cumin2002"
* 17:42 denisse@deploy2002: Finished deploy [librenms/librenms@f049593]: Upgrade LibreNMS to 24.12.0 - T384036 (duration: 00m 14s)
* 17:41 denisse@deploy2002: Started deploy [librenms/librenms@f049593]: Upgrade LibreNMS to 24.12.0 - T384036
* 17:40 denisse: Upgrading LibreNMS in production - T384036
* 17:40 hnowlan@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mw[2259,2263-2266].codfw.wmnet decommissioned, removing all IPs except the asset tag one - hnowlan@cumin2002"
* 17:36 hnowlan@cumin2002: START - Cookbook sre.dns.netbox
* 17:25 hnowlan@cumin2002: START - Cookbook sre.hosts.decommission for hosts mw[2259,2263-2266].codfw.wmnet
* 17:12 xcollazo@deploy2002: Finished deploy [airflow-dags/analytics@b0cd4df]: Deploy latest DAGs for 'analytics' Airflow instance. T366542. (duration: 00m 32s)
* 17:11 xcollazo@deploy2002: Started deploy [airflow-dags/analytics@b0cd4df]: Deploy latest DAGs for 'analytics' Airflow instance. T366542.
* 15:40 topranks: manually restarting netbox service on netbox1003
* 15:39 cmooney@cumin1002: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox
* 15:26 cmooney@cumin1002: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 15:20 cmooney@cumin1002: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 15:20 cmooney@cumin1002: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 15:13 moritzm: powercycle prometheus1006
* 14:48 marostegui@cumin1002: dbctl commit (dc=all): 'es1046 (re)pooling @ 100%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P72147 and previous config saved to /var/cache/conftool/dbconfig/20250117-144824-root.json
* 14:34 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2027.codfw.wmnet with OS bookworm
* 14:33 marostegui@cumin1002: dbctl commit (dc=all): 'es1046 (re)pooling @ 75%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P72146 and previous config saved to /var/cache/conftool/dbconfig/20250117-143318-root.json
* 14:18 marostegui@cumin1002: dbctl commit (dc=all): 'es1046 (re)pooling @ 50%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P72145 and previous config saved to /var/cache/conftool/dbconfig/20250117-141813-root.json
* 14:15 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2027.codfw.wmnet with reason: host reimage
* 14:12 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2027.codfw.wmnet with reason: host reimage
* 14:03 marostegui@cumin1002: dbctl commit (dc=all): 'es1046 (re)pooling @ 25%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P72144 and previous config saved to /var/cache/conftool/dbconfig/20250117-140308-root.json
* 14:01 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 14:01 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 14:00 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 14:00 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 13:51 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 13:50 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 13:48 marostegui@cumin1002: dbctl commit (dc=all): 'es1046 (re)pooling @ 10%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P72143 and previous config saved to /var/cache/conftool/dbconfig/20250117-134801-root.json
* 13:36 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 13:35 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 13:35 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 13:35 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 13:32 marostegui@cumin1002: dbctl commit (dc=all): 'es1046 (re)pooling @ 5%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P72142 and previous config saved to /var/cache/conftool/dbconfig/20250117-133256-root.json
* 13:17 marostegui@cumin1002: dbctl commit (dc=all): 'es1046 (re)pooling @ 4%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P72140 and previous config saved to /var/cache/conftool/dbconfig/20250117-131751-root.json
* 13:02 marostegui@cumin1002: dbctl commit (dc=all): 'es1046 (re)pooling @ 3%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P72139 and previous config saved to /var/cache/conftool/dbconfig/20250117-130245-root.json
* 12:47 marostegui@cumin1002: dbctl commit (dc=all): 'es1046 (re)pooling @ 2%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P72138 and previous config saved to /var/cache/conftool/dbconfig/20250117-124740-root.json
* 12:32 marostegui@cumin1002: dbctl commit (dc=all): 'es1046 (re)pooling @ 1%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P72137 and previous config saved to /var/cache/conftool/dbconfig/20250117-123235-root.json
* 12:31 marostegui@cumin1002: dbctl commit (dc=all): 'Add es1046 to dbctl depooled T382569', diff saved to https://phabricator.wikimedia.org/P72136 and previous config saved to /var/cache/conftool/dbconfig/20250117-123153-marostegui.json
* 12:25 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2141.codfw.wmnet with OS bookworm
* 12:07 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1111-1116].eqiad.wmnet
* 12:07 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1111-1116].eqiad.wmnet
* 12:02 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 12:02 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 12:01 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2141.codfw.wmnet with reason: host reimage
* 12:00 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 12:00 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 11:58 moritzm: installing Linux 6.1.124 on Bookworm hosts
* 11:57 jynus@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2141.codfw.wmnet with reason: host reimage
* 11:42 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti2027.codfw.wmnet with OS bookworm
* 11:41 jynus@cumin2002: START - Cookbook sre.hosts.reimage for host db2141.codfw.wmnet with OS bookworm
* 11:37 jynus@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2141.codfw.wmnet with reason: reimage
* 11:31 kamila@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker[1111-1116].eqiad.wmnet
* 11:31 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1111-1116].eqiad.wmnet
* 10:58 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2240-2241].codfw.wmnet
* 10:58 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2240-2241].codfw.wmnet
* 10:55 jelto: homer 'cr*codfw*' commit 'T377877'
* 10:54 jelto: homer 'lsw1-b3-codfw*' commit 'T377877'
* 10:51 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2241.codfw.wmnet with OS bookworm
* 10:31 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2241.codfw.wmnet with reason: host reimage
* 10:26 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2241.codfw.wmnet with reason: host reimage
* 10:19 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2240.codfw.wmnet with OS bookworm
* 10:08 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2241
* 10:08 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2241
* 10:08 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2241.codfw.wmnet with OS bookworm
* 10:08 jelto@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker2241.codfw.wmnet with OS bookworm
* 09:59 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2240.codfw.wmnet with reason: host reimage
* 09:55 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2240.codfw.wmnet with reason: host reimage
* 09:42 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2241
* 09:42 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2241
* 09:42 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2241
* 09:42 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2241.codfw.wmnet 158.16.192.10.in-addr.arpa 8.5.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 09:42 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2241.codfw.wmnet 158.16.192.10.in-addr.arpa 8.5.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 09:42 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:42 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2241 - jelto@cumin1002"
* 09:42 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2241 - jelto@cumin1002"
* 09:39 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 09:39 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2241
* 09:38 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2241.codfw.wmnet with OS bookworm
* 09:37 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2240
* 09:37 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2240
* 09:37 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2240
* 09:37 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2240.codfw.wmnet 157.16.192.10.in-addr.arpa 7.5.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 09:37 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2240.codfw.wmnet 157.16.192.10.in-addr.arpa 7.5.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 09:37 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:37 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2240 - jelto@cumin1002"
* 09:37 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2240 - jelto@cumin1002"
* 09:33 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 09:33 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2240
* 09:33 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2240.codfw.wmnet with OS bookworm
* 09:14 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2240.codfw.wmnet wikikube-worker2241.codfw.wmnet on all recursors
* 09:14 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2240.codfw.wmnet wikikube-worker2241.codfw.wmnet on all recursors
* 09:13 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2240 wikikube-worker2241 on all recursors
* 09:13 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2240 wikikube-worker2241 on all recursors
* 09:08 Emperor: depool / restart / repool ms-fe2010 T360913
* 09:08 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2311 to wikikube-worker2241
* 09:07 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2241
* 09:07 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2241
* 09:07 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:07 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2311 to wikikube-worker2241 - jelto@cumin1002"
* 09:06 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2311 to wikikube-worker2241 - jelto@cumin1002"
* 09:03 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 09:03 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2311 to wikikube-worker2241
* 09:01 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2310 to wikikube-worker2240
* 09:01 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2240
* 09:00 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2240
* 09:00 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:00 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2310 to wikikube-worker2240 - jelto@cumin1002"
* 08:59 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2310 to wikikube-worker2240 - jelto@cumin1002"
* 08:56 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 08:55 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2310 to wikikube-worker2240
* 08:35 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[2282,2310-2311].codfw.wmnet
* 08:33 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[2282,2310-2311].codfw.wmnet
* 07:48 marostegui@cumin1002: dbctl commit (dc=all): 'es1025 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72130 and previous config saved to /var/cache/conftool/dbconfig/20250117-074806-root.json
* 07:33 marostegui@cumin1002: dbctl commit (dc=all): 'es1025 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72129 and previous config saved to /var/cache/conftool/dbconfig/20250117-073301-root.json
* 07:17 marostegui@cumin1002: dbctl commit (dc=all): 'es1025 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72128 and previous config saved to /var/cache/conftool/dbconfig/20250117-071755-root.json
* 07:02 marostegui@cumin1002: dbctl commit (dc=all): 'es1025 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72127 and previous config saved to /var/cache/conftool/dbconfig/20250117-070250-root.json
* 06:47 marostegui@cumin1002: dbctl commit (dc=all): 'es1025 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72125 and previous config saved to /var/cache/conftool/dbconfig/20250117-064745-root.json
* 06:27 moritzm: installing rsync security regression updates
* 05:33 kartik@deploy2002: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
* 05:32 kartik@deploy2002: helmfile [eqiad] START helmfile.d/services/cxserver: apply
* 05:32 kartik@deploy2002: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
* 05:31 kartik@deploy2002: helmfile [codfw] START helmfile.d/services/cxserver: apply
* 05:27 kartik@deploy2002: helmfile [staging] DONE helmfile.d/services/cxserver: apply
* 05:26 kartik@deploy2002: helmfile [staging] START helmfile.d/services/cxserver: apply
* 00:23 zabe: zabe@deploy2002:/srv/mediawiki-staging$ mwscript-k8s -f -- createAndPromote.php --wiki=arbcom_zhwiki --sysop --bureaucrat --force ZhaoFJx
* 00:21 zabe: zabe@deploy2002:/srv/mediawiki-staging$ mwscript-k8s -f -- createAndPromote.php --wiki=arbcom_zhwiki ZhaoFJx REDACTED
* 00:18 zabe@deploy2002: Finished scap sync-world: Backport for gerrit:1112113Update interwiki cache, gerrit:1112111Update composer.lock, gerrit:1112114Activate arbcom_zhwiki (T380119) (duration: 11m 46s)
* 00:11 zabe@deploy2002: zabe: Continuing with sync
* 00:11 zabe@deploy2002: zabe: Backport for gerrit:1112113Update interwiki cache, gerrit:1112111Update composer.lock, gerrit:1112114Activate arbcom_zhwiki (T380119) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 00:07 zabe@deploy2002: Started scap sync-world: Backport for gerrit:1112113Update interwiki cache, gerrit:1112111Update composer.lock, gerrit:1112114Activate arbcom_zhwiki (T380119)
== 2025-01-16 ==
* 23:58 zabe@deploy2002: Finished scap sync-world: Creating arbcom_zhwiki (T380119) (duration: 12m 43s)
* 23:51 zabe@deploy2002: zabe: Continuing with sync
* 23:50 zabe@deploy2002: zabe: Creating arbcom_zhwiki (T380119) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 23:45 zabe@deploy2002: Started scap sync-world: Creating arbcom_zhwiki (T380119)
* 23:28 toyofuku@deploy2002: Finished scap sync-world: Backport for gerrit:1111715Beta: Update schemas in InitialiseSettings-labs.php (T382080) (duration: 11m 23s)
* 23:21 toyofuku@deploy2002: toyofuku, jdlrobson: Continuing with sync
* 23:21 toyofuku@deploy2002: toyofuku, jdlrobson: Backport for gerrit:1111715Beta: Update schemas in InitialiseSettings-labs.php (T382080) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 23:17 toyofuku@deploy2002: Started scap sync-world: Backport for gerrit:1111715Beta: Update schemas in InitialiseSettings-labs.php (T382080)
* 23:14 toyofuku@deploy2002: Finished scap sync-world: Backport for gerrit:1107964Stop expanding sections by default on Wiktionary (T376446) (duration: 11m 32s)
* 23:07 toyofuku@deploy2002: jdlrobson, toyofuku: Continuing with sync
* 23:07 toyofuku@deploy2002: jdlrobson, toyofuku: Backport for gerrit:1107964Stop expanding sections by default on Wiktionary (T376446) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 23:02 toyofuku@deploy2002: Started scap sync-world: Backport for gerrit:1107964Stop expanding sections by default on Wiktionary (T376446)
* 23:01 toyofuku@deploy2002: Finished scap sync-world: Backport for gerrit:1108135Remove `wgVectorStickyHeader` from InitialiseSettings.php (T332728) (duration: 12m 13s)
* 22:53 toyofuku@deploy2002: ksarabia, toyofuku: Continuing with sync
* 22:53 toyofuku@deploy2002: ksarabia, toyofuku: Backport for gerrit:1108135Remove `wgVectorStickyHeader` from InitialiseSettings.php (T332728) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 22:49 toyofuku@deploy2002: Started scap sync-world: Backport for gerrit:1108135Remove `wgVectorStickyHeader` from InitialiseSettings.php (T332728)
* 22:45 toyofuku@deploy2002: Finished scap sync-world: Backport for gerrit:1112074Configure streams for web empty search AB test (T380926) (duration: 16m 47s)
* 22:38 toyofuku@deploy2002: bwang, toyofuku: Continuing with sync
* 22:33 toyofuku@deploy2002: bwang, toyofuku: Backport for gerrit:1112074Configure streams for web empty search AB test (T380926) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 22:28 toyofuku@deploy2002: Started scap sync-world: Backport for gerrit:1112074Configure streams for web empty search AB test (T380926)
* 22:15 cjming@deploy2002: Finished scap sync-world: Backport for gerrit:1111630Analytics - Set parameters to Refine content history reconcile (T369845) (duration: 12m 37s)
* 22:08 cjming@deploy2002: aqu, cjming: Continuing with sync
* 22:07 cjming@deploy2002: aqu, cjming: Backport for gerrit:1111630Analytics - Set parameters to Refine content history reconcile (T369845) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 22:03 cjming@deploy2002: Started scap sync-world: Backport for gerrit:1111630Analytics - Set parameters to Refine content history reconcile (T369845)
* 22:01 cjming@deploy2002: Finished scap sync-world: Backport for gerrit:1111262testwiki: Enable MetricsPlatform stream config fetching and merging (T381964) (duration: 13m 10s)
* 22:01 volans: uploaded cumin_5.0.0 to apt.wikimedia.org bullseye-wikimedia
* 21:54 cjming@deploy2002: phuedx, cjming: Continuing with sync
* 21:54 cjming@deploy2002: phuedx, cjming: Backport for gerrit:1111262testwiki: Enable MetricsPlatform stream config fetching and merging (T381964) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:49 cdanis@deploy2002: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
* 21:49 cdanis@deploy2002: helmfile [staging] START helmfile.d/services/chart-renderer: apply
* 21:48 cjming@deploy2002: Started scap sync-world: Backport for gerrit:1111262testwiki: Enable MetricsPlatform stream config fetching and merging (T381964)
* 21:48 cdanis@deploy2002: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
* 21:47 cdanis@deploy2002: helmfile [staging] START helmfile.d/services/chart-renderer: apply
* 21:47 cjming@deploy2002: Finished scap sync-world: Backport for gerrit:1111261Enable MetricsPlatform extension everywhere (T381964) (duration: 11m 43s)
* 21:40 cjming@deploy2002: cjming, phuedx: Continuing with sync
* 21:40 cjming@deploy2002: cjming, phuedx: Backport for gerrit:1111261Enable MetricsPlatform extension everywhere (T381964) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:35 cjming@deploy2002: Started scap sync-world: Backport for gerrit:1111261Enable MetricsPlatform extension everywhere (T381964)
* 21:33 cjming@deploy2002: Finished scap sync-world: Backport for gerrit:1111260Beta Cluster: Update MetricsPlatform extension config (T381964) (duration: 12m 55s)
* 21:26 cjming@deploy2002: cjming, phuedx: Continuing with sync
* 21:26 cjming@deploy2002: cjming, phuedx: Backport for gerrit:1111260Beta Cluster: Update MetricsPlatform extension config (T381964) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:20 cjming@deploy2002: Started scap sync-world: Backport for gerrit:1111260Beta Cluster: Update MetricsPlatform extension config (T381964)
* 21:17 jsn@deploy2002: Finished scap sync-world: Backport for gerrit:1111350Increase Nuke max age to 90 days (attempt 2) (T380846) (duration: 13m 12s)
* 21:09 jsn@deploy2002: jsn, chlod: Continuing with sync
* 21:08 jsn@deploy2002: jsn, chlod: Backport for gerrit:1111350Increase Nuke max age to 90 days (attempt 2) (T380846) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:04 jsn@deploy2002: Started scap sync-world: Backport for gerrit:1111350Increase Nuke max age to 90 days (attempt 2) (T380846)
* 20:59 catrope@deploy2002: helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply
* 20:58 catrope@deploy2002: helmfile [codfw] START helmfile.d/services/chart-renderer: apply
* 20:58 catrope@deploy2002: helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply
* 20:57 catrope@deploy2002: helmfile [eqiad] START helmfile.d/services/chart-renderer: apply
* 20:51 catrope@deploy2002: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
* 20:50 catrope@deploy2002: helmfile [staging] START helmfile.d/services/chart-renderer: apply
* 20:28 kamila@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker[1111-1116].eqiad.wmnet
* 20:28 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1111-1116].eqiad.wmnet
* 20:27 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1114.eqiad.wmnet with OS bookworm
* 20:08 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1114.eqiad.wmnet with reason: host reimage
* 20:04 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1114.eqiad.wmnet with reason: host reimage
* 19:48 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1114
* 19:48 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1114
* 19:48 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1114.eqiad.wmnet with OS bookworm
* 19:19 brennen@deploy2002: rebuilt and synchronized wikiversions files: group2 to 1.44.0-wmf.12 refs T382363
* 19:06 brennen: 1.44.0-wmf.12 train (T382363): no current blockers and logs calm, rolling to all wikis.
* 18:40 bd808@deploy2002: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 18:39 bd808@deploy2002: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 18:38 bd808@deploy2002: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 18:38 bd808@deploy2002: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 18:37 bd808@deploy2002: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 18:37 bd808@deploy2002: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 18:28 swfrench@deploy2002: Finished scap sync-world: Deployment to switch next release files to 8.1 - T377040 (duration: 03m 50s)
* 18:24 swfrench@deploy2002: Started scap sync-world: Deployment to switch next release files to 8.1 - T377040
* 18:22 cmooney@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 138398
* 18:22 cmooney@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 138398
* 17:54 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1113.eqiad.wmnet with OS bookworm
* 17:50 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1112.eqiad.wmnet with OS bookworm
* 17:47 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1116.eqiad.wmnet with OS bookworm
* 17:44 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1115.eqiad.wmnet with OS bookworm
* 17:40 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1111.eqiad.wmnet with OS bookworm
* 17:35 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1113.eqiad.wmnet with reason: host reimage
* 17:34 zabe@deploy2002: Finished scap sync-world: T380119 (duration: 10m 59s)
* 17:31 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1112.eqiad.wmnet with reason: host reimage
* 17:28 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1116.eqiad.wmnet with reason: host reimage
* 17:25 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1115.eqiad.wmnet with reason: host reimage
* 17:23 zabe@deploy2002: Started scap sync-world: T380119
* 17:23 zabe@deploy2002: sync-world aborted: T380119 (duration: 01m 16s)
* 17:22 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1111.eqiad.wmnet with reason: host reimage
* 17:22 zabe@deploy2002: Started scap sync-world: T380119
* 17:19 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1116.eqiad.wmnet with reason: host reimage
* 17:19 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1113.eqiad.wmnet with reason: host reimage
* 17:19 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1115.eqiad.wmnet with reason: host reimage
* 17:18 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1112.eqiad.wmnet with reason: host reimage
* 17:18 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1111.eqiad.wmnet with reason: host reimage
* 17:14 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1114.eqiad.wmnet on all recursors
* 17:14 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1114.eqiad.wmnet on all recursors
* 17:03 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1116
* 17:03 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1116
* 17:03 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1116.eqiad.wmnet with OS bookworm
* 17:03 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1115
* 17:03 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1115
* 17:03 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1115.eqiad.wmnet with OS bookworm
* 17:03 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1113
* 17:03 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1113
* 17:03 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1113.eqiad.wmnet with OS bookworm
* 17:03 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1112
* 17:02 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1112
* 17:02 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1112.eqiad.wmnet with OS bookworm
* 17:02 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1111
* 17:02 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1111
* 17:02 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1111.eqiad.wmnet with OS bookworm
* 16:59 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1111.eqiad.wmnet wikikube-worker1112.eqiad.wmnet wikikube-worker1113.eqiad.wmnet wikikube-worker1114.eqiad.wmnet wikikube-worker1115.eqiad.wmnet wikikube-worker1116.eqiad.wmnet on all recursors
* 16:59 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1111.eqiad.wmnet wikikube-worker1112.eqiad.wmnet wikikube-worker1113.eqiad.wmnet wikikube-worker1114.eqiad.wmnet wikikube-worker1115.eqiad.wmnet wikikube-worker1116.eqiad.wmnet on all recursors
* 16:59 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1463 to wikikube-worker1116
* 16:58 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1116
* 16:56 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1116
* 16:56 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:56 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1463 to wikikube-worker1116 - kamila@cumin1002"
* 16:56 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1463 to wikikube-worker1116 - kamila@cumin1002"
* 16:54 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1462 to wikikube-worker1115
* 16:53 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1115
* 16:52 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 16:52 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1115
* 16:52 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:52 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1462 to wikikube-worker1115 - kamila@cumin1002"
* 16:52 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1462 to wikikube-worker1115 - kamila@cumin1002"
* 16:49 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1463 to wikikube-worker1116
* 16:49 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1461 to wikikube-worker1114
* 16:48 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1114
* 16:48 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 16:47 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1114
* 16:47 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:47 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1461 to wikikube-worker1114 - kamila@cumin1002"
* 16:47 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1461 to wikikube-worker1114 - kamila@cumin1002"
* 16:45 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1462 to wikikube-worker1115
* 16:44 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1460 to wikikube-worker1113
* 16:44 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1113
* 16:43 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 16:43 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1113
* 16:43 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:43 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1460 to wikikube-worker1113 - kamila@cumin1002"
* 16:42 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1460 to wikikube-worker1113 - kamila@cumin1002"
* 16:39 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1449 to wikikube-worker1112
* 16:39 topranks: manually restarting netbox service on netbox1003 to update interface validator
* 16:39 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1112
* 16:38 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 16:38 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1461 to wikikube-worker1114
* 16:38 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1112
* 16:38 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:38 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1449 to wikikube-worker1112 - kamila@cumin1002"
* 16:38 cmooney@cumin1002: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox
* 16:37 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 16:37 cmooney@cumin1002: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 16:37 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1449 to wikikube-worker1112 - kamila@cumin1002"
* 16:37 cmooney@cumin1002: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 16:37 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 16:36 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1460 to wikikube-worker1113
* 16:36 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1448 to wikikube-worker1111
* 16:36 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 16:35 cmooney@cumin1002: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 16:35 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1111
* 16:35 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 16:34 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1111
* 16:34 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:34 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1448 to wikikube-worker1111 - kamila@cumin1002"
* 16:34 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 16:34 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1448 to wikikube-worker1111 - kamila@cumin1002"
* 16:33 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1449 to wikikube-worker1112
* 16:29 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 16:28 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1448 to wikikube-worker1111
* 16:25 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[1448-1449,1460-1463].eqiad.wmnet
* 16:22 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[1448-1449,1460-1463].eqiad.wmnet
* 16:21 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 16:21 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 16:19 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 16:19 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 16:05 ladsgroup@deploy2002: Finished scap sync-world: Backport for gerrit:1112047dbconfig: Order json output entries (duration: 18m 52s)
* 15:58 ladsgroup@deploy2002: ladsgroup: Continuing with sync
* 15:54 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1107-1110].eqiad.wmnet
* 15:54 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1107-1110].eqiad.wmnet
* 15:53 ladsgroup@deploy2002: ladsgroup: Backport for gerrit:1112047dbconfig: Order json output entries synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 15:52 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1107.eqiad.wmnet with OS bookworm
* 15:51 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: sync
* 15:51 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/eventstreams: sync
* 15:46 ladsgroup@deploy2002: Started scap sync-world: Backport for gerrit:1112047dbconfig: Order json output entries
* 15:44 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1111949Check known-good regex patterns directly (T380751) (duration: 14m 32s)
* 15:37 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde: Continuing with sync
* 15:36 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde: Backport for gerrit:1111949Check known-good regex patterns directly (T380751) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 15:33 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1107.eqiad.wmnet with reason: host reimage
* 15:33 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2236-2239].codfw.wmnet
* 15:33 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2236-2239].codfw.wmnet
* 15:32 jelto: homer 'cr*codfw*' commit 'T377877'
* 15:31 jelto: homer 'lsw1-c3-codfw*' commit 'T377877'
* 15:30 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1107.eqiad.wmnet with reason: host reimage
* 15:29 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2238.codfw.wmnet with OS bookworm
* 15:29 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1111949Check known-good regex patterns directly (T380751)
* 15:24 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1110.eqiad.wmnet with OS bookworm
* 15:20 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1109.eqiad.wmnet with OS bookworm
* 15:19 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72124 and previous config saved to /var/cache/conftool/dbconfig/20250116-151950-root.json
* 15:18 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1112019Disable distinct-values constraint checks on Commons (T369079) (duration: 10m 51s)
* 15:14 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1107
* 15:14 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1107
* 15:14 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1107.eqiad.wmnet with OS bookworm
* 15:13 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde: Continuing with sync
* 15:13 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde: Backport for gerrit:1112019Disable distinct-values constraint checks on Commons (T369079) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 15:13 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1108.eqiad.wmnet with OS bookworm
* 15:09 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2238.codfw.wmnet with reason: host reimage
* 15:07 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1112019Disable distinct-values constraint checks on Commons (T369079)
* 15:05 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1110.eqiad.wmnet with reason: host reimage
* 15:04 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72123 and previous config saved to /var/cache/conftool/dbconfig/20250116-150446-root.json
* 15:03 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2238.codfw.wmnet with reason: host reimage
* 15:02 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1109.eqiad.wmnet with reason: host reimage
* 14:59 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2239.codfw.wmnet with OS bookworm
* 14:57 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1110.eqiad.wmnet with reason: host reimage
* 14:57 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1109.eqiad.wmnet with reason: host reimage
* 14:53 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1108.eqiad.wmnet with reason: host reimage
* 14:51 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1112015Add z-index to `.tux-more-notices` (T383669) (duration: 22m 26s)
* 14:50 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1108.eqiad.wmnet with reason: host reimage
* 14:49 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72122 and previous config saved to /var/cache/conftool/dbconfig/20250116-144940-root.json
* 14:46 lucaswerkmeister-wmde@deploy2002: abi, lucaswerkmeister-wmde: Continuing with sync
* 14:45 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2238
* 14:45 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2238
* 14:45 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2238.codfw.wmnet with OS bookworm
* 14:45 jelto@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker2238.codfw.wmnet with OS bookworm
* 14:41 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1110
* 14:41 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1110
* 14:41 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1110.eqiad.wmnet with OS bookworm
* 14:41 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1109
* 14:41 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1109
* 14:41 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1109.eqiad.wmnet with OS bookworm
* 14:39 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2239.codfw.wmnet with reason: host reimage
* 14:36 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2239.codfw.wmnet with reason: host reimage
* 14:36 lucaswerkmeister-wmde@deploy2002: abi, lucaswerkmeister-wmde: Backport for gerrit:1112015Add z-index to `.tux-more-notices` (T383669) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:35 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2237.codfw.wmnet with OS bookworm
* 14:34 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1108
* 14:34 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1108
* 14:34 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1108.eqiad.wmnet with OS bookworm
* 14:34 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72121 and previous config saved to /var/cache/conftool/dbconfig/20250116-143435-root.json
* 14:34 kamila@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1110.eqiad.wmnet with OS bookworm
* 14:33 kamila@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1109.eqiad.wmnet with OS bookworm
* 14:33 kamila@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1108.eqiad.wmnet with OS bookworm
* 14:30 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1107.eqiad.wmnet on all recursors
* 14:29 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1107.eqiad.wmnet on all recursors
* 14:29 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1112015Add z-index to `.tux-more-notices` (T383669)
* 14:28 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp
* 14:27 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1111325Turn on Parsoid Read Views on test2wiki (T378645) (duration: 20m 22s)
* 14:27 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1110
* 14:27 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1110
* 14:27 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1109
* 14:27 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1109
* 14:27 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1110.eqiad.wmnet with OS bookworm
* 14:26 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1108
* 14:26 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1108
* 14:26 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1109.eqiad.wmnet with OS bookworm
* 14:26 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1108.eqiad.wmnet with OS bookworm
* 14:26 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2236.codfw.wmnet with OS bookworm
* 14:23 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1107.eqiad.wmnet wikikube-worker1108.eqiad.wmnet wikikube-worker1109.eqiad.wmnet wikikube-worker1110.eqiad.wmnet on all recursors
* 14:23 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1107.eqiad.wmnet wikikube-worker1108.eqiad.wmnet wikikube-worker1109.eqiad.wmnet wikikube-worker1110.eqiad.wmnet on all recursors
* 14:22 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1441 to wikikube-worker1109
* 14:22 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1109
* 14:21 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1109
* 14:21 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:20 lucaswerkmeister-wmde@deploy2002: ssastry, lucaswerkmeister-wmde: Continuing with sync
* 14:18 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 14:18 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1442 to wikikube-worker1110
* 14:18 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2239
* 14:18 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2239
* 14:18 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2239
* 14:18 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2239.codfw.wmnet 116.32.192.10.in-addr.arpa 6.1.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:18 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2239.codfw.wmnet 116.32.192.10.in-addr.arpa 6.1.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:18 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:17 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1110
* 14:17 lucaswerkmeister-wmde@deploy2002: ssastry, lucaswerkmeister-wmde: Backport for gerrit:1111325Turn on Parsoid Read Views on test2wiki (T378645) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:16 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1110
* 14:16 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:16 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1442 to wikikube-worker1110 - kamila@cumin1002"
* 14:15 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 14:15 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1442 to wikikube-worker1110 - kamila@cumin1002"
* 14:15 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72120 and previous config saved to /var/cache/conftool/dbconfig/20250116-141530-root.json
* 14:15 jelto@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 14:14 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2237.codfw.wmnet with reason: host reimage
* 14:13 marostegui@dns1006: END - running authdns-update
* 14:11 marostegui@dns1006: START - running authdns-update
* 14:10 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2237.codfw.wmnet with reason: host reimage
* 14:10 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 14:09 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp
* 14:08 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp
* 14:07 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1442 to wikikube-worker1110
* 14:07 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1441 to wikikube-worker1109
* 14:07 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1440 to wikikube-worker1108
* 14:06 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1111325Turn on Parsoid Read Views on test2wiki (T378645)
* 14:06 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2236.codfw.wmnet with reason: host reimage
* 14:06 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1108
* 14:05 marostegui@cumin1002: dbctl commit (dc=all): 'Pool pc6 eqiad and codfw dbmaint T383234', diff saved to https://phabricator.wikimedia.org/P72119 and previous config saved to /var/cache/conftool/dbconfig/20250116-140523-marostegui.json
* 14:05 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 14:05 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1108
* 14:05 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:04 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2239
* 14:04 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2239.codfw.wmnet with OS bookworm
* 14:02 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 14:02 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2236.codfw.wmnet with reason: host reimage
* 14:02 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2238
* 14:02 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2238
* 14:01 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2238
* 14:01 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2238.codfw.wmnet 115.32.192.10.in-addr.arpa 5.1.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:01 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2238.codfw.wmnet 115.32.192.10.in-addr.arpa 5.1.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:01 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:01 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2238 - jelto@cumin1002"
* 14:01 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2238 - jelto@cumin1002"
* 13:57 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:57 jynus@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts dbprov2002.codfw.wmnet
* 13:57 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:57 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbprov2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002"
* 13:57 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbprov2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002"
* 13:56 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2238
* 13:56 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2238.codfw.wmnet with OS bookworm
* 13:53 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2237
* 13:53 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2237
* 13:53 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 13:52 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2237
* 13:52 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2237.codfw.wmnet 113.32.192.10.in-addr.arpa 3.1.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:52 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2237.codfw.wmnet 113.32.192.10.in-addr.arpa 3.1.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:52 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:52 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2237 - jelto@cumin1002"
* 13:52 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2237 - jelto@cumin1002"
* 13:50 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1439 to wikikube-worker1107
* 13:49 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp
* 13:49 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1440 to wikikube-worker1108
* 13:49 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1107
* 13:48 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:48 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1107
* 13:48 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:48 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1439 to wikikube-worker1107 - kamila@cumin1002"
* 13:48 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1439 to wikikube-worker1107 - kamila@cumin1002"
* 13:45 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2237
* 13:45 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2237.codfw.wmnet with OS bookworm
* 13:44 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2236
* 13:44 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2236
* 13:44 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 13:44 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2236
* 13:44 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2236.codfw.wmnet 112.32.192.10.in-addr.arpa 2.1.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:44 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2236.codfw.wmnet 112.32.192.10.in-addr.arpa 2.1.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:44 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:44 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2236 - jelto@cumin1002"
* 13:43 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2236 - jelto@cumin1002"
* 13:43 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1439 to wikikube-worker1107
* 13:42 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[1439-1442].eqiad.wmnet
* 13:40 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:40 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2236
* 13:40 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2236.codfw.wmnet with OS bookworm
* 13:39 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[1439-1442].eqiad.wmnet
* 13:38 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2236.codfw.wmnet wikikube-worker2237.codfw.wmnet wikikube-worker2238.codfw.wmnet wikikube-worker2239.codfw.wmnet on all recursors
* 13:38 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2236.codfw.wmnet wikikube-worker2237.codfw.wmnet wikikube-worker2238.codfw.wmnet wikikube-worker2239.codfw.wmnet on all recursors
* 13:37 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2338 to wikikube-worker2239
* 13:37 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2239
* 13:36 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2239
* 13:36 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:36 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2338 to wikikube-worker2239 - jelto@cumin1002"
* 13:36 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts dbprov2002.codfw.wmnet
* 13:36 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2338 to wikikube-worker2239 - jelto@cumin1002"
* 13:35 jynus@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts dbprov2002.codfw.wmnet
* 13:35 jynus@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 13:32 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:32 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2338 to wikikube-worker2239
* 13:30 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2337 to wikikube-worker2238
* 13:30 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2238
* 13:29 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2238
* 13:29 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:29 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2337 to wikikube-worker2238 - jelto@cumin1002"
* 13:29 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2337 to wikikube-worker2238 - jelto@cumin1002"
* 13:26 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:25 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2337 to wikikube-worker2238
* 13:24 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2336 to wikikube-worker2237
* 13:23 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2237
* 13:23 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2237
* 13:23 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:23 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2336 to wikikube-worker2237 - jelto@cumin1002"
* 13:23 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2336 to wikikube-worker2237 - jelto@cumin1002"
* 13:18 hashar: Upgraded Gerrit to 3.10.4
* 13:12 hashar@deploy2002: Finished deploy [gerrit/gerrit@5c2347d]: Gerrit to 3.10.4 - T383597 (duration: 00m 10s)
* 13:12 hashar@deploy2002: Started deploy [gerrit/gerrit@5c2347d]: Gerrit to 3.10.4 - T383597
* 13:09 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:08 hashar@deploy2002: Finished deploy [gerrit/gerrit@5c2347d]: Gerrit to 3.10.4 - T383597 (duration: 00m 08s)
* 13:08 hashar@deploy2002: Started deploy [gerrit/gerrit@5c2347d]: Gerrit to 3.10.4 - T383597
* 13:08 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2336 to wikikube-worker2237
* 13:04 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 12:56 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts dbprov2002.codfw.wmnet
* 12:52 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dbprov2001.codfw.wmnet
* 12:52 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2335 to wikikube-worker2236
* 12:50 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2236
* 12:50 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 12:50 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2236
* 12:50 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:50 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2335 to wikikube-worker2236 - jelto@cumin1002"
* 12:49 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2335 to wikikube-worker2236 - jelto@cumin1002"
* 12:45 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 12:45 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2335 to wikikube-worker2236
* 12:44 marostegui@dns1006: END - running authdns-update
* 12:42 marostegui@dns1006: START - running authdns-update
* 12:42 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[2335-2338].codfw.wmnet
* 12:40 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[2335-2338].codfw.wmnet
* 12:39 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts dbprov2001.codfw.wmnet
* 12:38 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dbprov1002.eqiad.wmnet
* 12:38 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:38 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbprov1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002"
* 12:36 marostegui@cumin1002: dbctl commit (dc=all): 'Promote pc1014 to pc4 eqiad master dbmaint and enable pc4 back in eqiad and codfw T383398', diff saved to https://phabricator.wikimedia.org/P72118 and previous config saved to /var/cache/conftool/dbconfig/20250116-123656-marostegui.json
* 12:33 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbprov1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002"
* 12:32 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on pc2014.codfw.wmnet,pc[1014,1016].eqiad.wmnet with reason: reorganizing pc4
* 12:31 marostegui@cumin1002: dbctl commit (dc=all): 'Depool pc4 eqiad and codfw dbmaint T383398', diff saved to https://phabricator.wikimedia.org/P72117 and previous config saved to /var/cache/conftool/dbconfig/20250116-123129-marostegui.json
* 12:29 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 12:27 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2232-2235].codfw.wmnet
* 12:27 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2232-2235].codfw.wmnet
* 12:24 jelto: homer 'cr*codfw*' commit 'T377877'
* 12:23 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts dbprov1002.eqiad.wmnet
* 12:23 jelto: homer 'lsw1-c6-codfw*' commit 'T377877'
* 12:23 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dbprov1001.eqiad.wmnet
* 12:23 jynus@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:23 jynus@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbprov1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002"
* 12:22 jynus@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbprov1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002"
* 12:22 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2235.codfw.wmnet with OS bookworm
* 12:20 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and A:cp
* 12:17 jynus@cumin1002: START - Cookbook sre.dns.netbox
* 12:14 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2234.codfw.wmnet with OS bookworm
* 12:09 jynus@cumin1002: START - Cookbook sre.hosts.decommission for hosts dbprov1001.eqiad.wmnet
* 12:06 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2233.codfw.wmnet with OS bookworm
* 12:02 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2235.codfw.wmnet with reason: host reimage
* 12:01 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2232.codfw.wmnet with OS bookworm
* 11:56 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2235.codfw.wmnet with reason: host reimage
* 11:54 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and A:cp
* 11:53 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2234.codfw.wmnet with reason: host reimage
* 11:50 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2234.codfw.wmnet with reason: host reimage
* 11:49 kart_: Updated cxserver to 2025-01-16-103443-production (T383854, T377966)
* 11:47 kartik@deploy2002: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
* 11:46 kartik@deploy2002: helmfile [eqiad] START helmfile.d/services/cxserver: apply
* 11:46 kartik@deploy2002: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
* 11:46 kartik@deploy2002: helmfile [codfw] START helmfile.d/services/cxserver: apply
* 11:45 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2233.codfw.wmnet with reason: host reimage
* 11:42 kartik@deploy2002: helmfile [staging] DONE helmfile.d/services/cxserver: apply
* 11:42 kartik@deploy2002: helmfile [staging] START helmfile.d/services/cxserver: apply
* 11:42 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2232.codfw.wmnet with reason: host reimage
* 11:40 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2233.codfw.wmnet with reason: host reimage
* 11:38 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2232.codfw.wmnet with reason: host reimage
* 11:38 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2235
* 11:38 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2235
* 11:38 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2235.codfw.wmnet with OS bookworm
* 11:37 jelto@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2235.codfw.wmnet
* 11:37 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2235.codfw.wmnet with OS bullseye
* 11:32 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2234
* 11:32 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2234
* 11:32 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2234.codfw.wmnet with OS bookworm
* 11:30 jelto@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2234.codfw.wmnet
* 11:30 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2234.codfw.wmnet with OS bullseye
* 11:29 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2149.codfw.wmnet
* 11:22 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2233
* 11:22 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2233
* 11:22 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2149', diff saved to https://phabricator.wikimedia.org/P72114 and previous config saved to /var/cache/conftool/dbconfig/20250116-112235-marostegui.json
* 11:22 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2233.codfw.wmnet with OS bookworm
* 11:22 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2149.codfw.wmnet
* 11:21 jelto@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2233.codfw.wmnet
* 11:21 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2233.codfw.wmnet with OS bullseye
* 11:20 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2232
* 11:20 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2232
* 11:20 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2232.codfw.wmnet with OS bookworm
* 11:19 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host wikikube-worker2232.codfw.wmnet with OS bookworm
* 11:19 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2232.codfw.wmnet with OS bookworm
* 11:17 jelto@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2232.codfw.wmnet
* 11:17 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2235.codfw.wmnet with reason: host reimage
* 11:15 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2232.codfw.wmnet with OS bullseye
* 11:13 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2235.codfw.wmnet with reason: host reimage
* 11:10 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2234.codfw.wmnet with reason: host reimage
* 11:06 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2234.codfw.wmnet with reason: host reimage
* 11:05 marostegui@cumin1002: dbctl commit (dc=all): 'es1045 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72113 and previous config saved to /var/cache/conftool/dbconfig/20250116-110538-root.json
* 11:01 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2233.codfw.wmnet with reason: host reimage
* 11:01 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2132.codfw.wmnet
* 11:01 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:58 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 10:58 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2235
* 10:58 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2235
* 10:58 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2235
* 10:57 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2235.codfw.wmnet 203.32.192.10.in-addr.arpa 3.0.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:57 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2235.codfw.wmnet 203.32.192.10.in-addr.arpa 3.0.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:57 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:57 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2235 - jelto@cumin1002"
* 10:57 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2233.codfw.wmnet with reason: host reimage
* 10:57 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2235 - jelto@cumin1002"
* 10:54 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2232.codfw.wmnet with reason: host reimage
* 10:53 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2132.codfw.wmnet
* 10:53 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 10:53 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2235
* 10:53 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2235.codfw.wmnet with OS bullseye
* 10:53 jelto@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2235.codfw.wmnet
* 10:50 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2234
* 10:50 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2234
* 10:50 marostegui@cumin1002: dbctl commit (dc=all): 'es1045 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72112 and previous config saved to /var/cache/conftool/dbconfig/20250116-105032-root.json
* 10:50 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2232.codfw.wmnet with reason: host reimage
* 10:50 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2234
* 10:50 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2234.codfw.wmnet 202.32.192.10.in-addr.arpa 2.0.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:50 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2234.codfw.wmnet 202.32.192.10.in-addr.arpa 2.0.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:50 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:49 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2234 - jelto@cumin1002"
* 10:49 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2234 - jelto@cumin1002"
* 10:46 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 10:46 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2234
* 10:46 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2234.codfw.wmnet with OS bullseye
* 10:45 jelto@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2234.codfw.wmnet
* 10:42 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2233
* 10:42 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2233
* 10:42 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2233
* 10:42 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2233.codfw.wmnet 201.32.192.10.in-addr.arpa 1.0.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:42 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2233.codfw.wmnet 201.32.192.10.in-addr.arpa 1.0.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:42 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:42 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2233 - jelto@cumin1002"
* 10:41 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2233 - jelto@cumin1002"
* 10:38 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 10:38 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2233
* 10:38 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2233.codfw.wmnet with OS bullseye
* 10:37 jelto@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2233.codfw.wmnet
* 10:35 marostegui@cumin1002: dbctl commit (dc=all): 'es1045 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72111 and previous config saved to /var/cache/conftool/dbconfig/20250116-103527-root.json
* 10:35 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2232
* 10:34 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2232
* 10:34 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2232
* 10:34 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2232.codfw.wmnet 200.32.192.10.in-addr.arpa 0.0.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:34 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2232.codfw.wmnet 200.32.192.10.in-addr.arpa 0.0.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:34 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:34 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2232 - jelto@cumin1002"
* 10:34 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2232 - jelto@cumin1002"
* 10:31 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 10:30 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2232
* 10:30 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2232.codfw.wmnet with OS bullseye
* 10:30 jelto@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2232.codfw.wmnet
* 10:28 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 10:27 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 10:26 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2232.codfw.wmnet wikikube-worker2233.codfw.wmnet wikikube-worker2234.codfw.wmnet wikikube-worker2235.codfw.wmnet on all recursors
* 10:26 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2232.codfw.wmnet wikikube-worker2233.codfw.wmnet wikikube-worker2234.codfw.wmnet wikikube-worker2235.codfw.wmnet on all recursors
* 10:26 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2353 to wikikube-worker2235
* 10:25 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2235
* 10:25 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2235
* 10:25 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:25 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2353 to wikikube-worker2235 - jelto@cumin1002"
* 10:25 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2353 to wikikube-worker2235 - jelto@cumin1002"
* 10:21 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 10:20 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2353 to wikikube-worker2235
* 10:20 marostegui@cumin1002: dbctl commit (dc=all): 'es1045 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72110 and previous config saved to /var/cache/conftool/dbconfig/20250116-102021-root.json
* 10:19 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.rename (exit_code=99) from mw2353 to wikikube-worker2235
* 10:19 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2353 to wikikube-worker2235
* 10:18 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2352 to wikikube-worker2234
* 10:17 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2234
* 10:17 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2234
* 10:17 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:17 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2352 to wikikube-worker2234 - jelto@cumin1002"
* 10:17 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2352 to wikikube-worker2234 - jelto@cumin1002"
* 10:13 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 10:13 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2352 to wikikube-worker2234
* 10:12 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2351 to wikikube-worker2233
* 10:11 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2233
* 10:11 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2233
* 10:11 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:11 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2351 to wikikube-worker2233 - jelto@cumin1002"
* 10:10 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2351 to wikikube-worker2233 - jelto@cumin1002"
* 10:07 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 10:06 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2351 to wikikube-worker2233
* 10:06 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2350 to wikikube-worker2232
* 10:05 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2232
* 10:05 marostegui@cumin1002: dbctl commit (dc=all): 'es1045 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72109 and previous config saved to /var/cache/conftool/dbconfig/20250116-100516-root.json
* 10:05 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2232
* 10:05 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:05 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2350 to wikikube-worker2232 - jelto@cumin1002"
* 10:04 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2350 to wikikube-worker2232 - jelto@cumin1002"
* 10:01 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 10:01 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 10:00 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2350 to wikikube-worker2232
* 10:00 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 09:58 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[2350-2353].codfw.wmnet
* 09:56 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[2350-2353].codfw.wmnet
* 09:54 marostegui@cumin1002: dbctl commit (dc=all): 'Promote pc2015 to pc5 codfw master dbmaint and enable pc5 back in eqiad and codfw T383398', diff saved to https://phabricator.wikimedia.org/P72107 and previous config saved to /var/cache/conftool/dbconfig/20250116-095431-marostegui.json
* 09:50 marostegui@cumin1002: dbctl commit (dc=all): 'es1045 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P72106 and previous config saved to /var/cache/conftool/dbconfig/20250116-095011-root.json
* 09:45 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on pc[2015,2017].codfw.wmnet,pc[1015,1017].eqiad.wmnet with reason: reorganizing pc5
* 09:44 marostegui@cumin1002: dbctl commit (dc=all): 'Depool pc5 eqiad and codfw dbmaint T383398', diff saved to https://phabricator.wikimedia.org/P72104 and previous config saved to /var/cache/conftool/dbconfig/20250116-094439-marostegui.json
* 09:35 marostegui@cumin1002: dbctl commit (dc=all): 'es1045 (re)pooling @ 4%: Repooling', diff saved to https://phabricator.wikimedia.org/P72103 and previous config saved to /var/cache/conftool/dbconfig/20250116-093505-root.json
* 09:31 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72102 and previous config saved to /var/cache/conftool/dbconfig/20250116-093113-root.json
* 09:29 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2228-2231].codfw.wmnet
* 09:29 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2228-2231].codfw.wmnet
* 09:27 jelto: homer 'cr*codfw*' commit 'T377877'
* 09:26 jelto: homer 'lsw1-c6-codfw*' commit 'T377877'
* 09:25 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2231.codfw.wmnet with OS bookworm
* 09:23 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2230.codfw.wmnet with OS bookworm
* 09:20 marostegui@cumin1002: dbctl commit (dc=all): 'es1045 (re)pooling @ 3%: Repooling', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20250116-091956-root.json
* 09:16 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72100 and previous config saved to /var/cache/conftool/dbconfig/20250116-091607-root.json
* 09:14 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2229.codfw.wmnet with OS bookworm
* 09:12 hashar: UTC morning backport window completed.
* 09:08 hashar@deploy2002: Finished scap sync-world: Backport for gerrit:1111661Add dso and thq to wmgExtraLanguageNames (T383785) (duration: 12m 46s)
* 09:06 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2231.codfw.wmnet with reason: host reimage
* 09:05 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2228.codfw.wmnet with OS bookworm
* 09:04 marostegui@cumin1002: dbctl commit (dc=all): 'es1045 (re)pooling @ 2%: Repooling', diff saved to https://phabricator.wikimedia.org/P72099 and previous config saved to /var/cache/conftool/dbconfig/20250116-090451-root.json
* 09:03 hashar@deploy2002: anzx, hashar: Continuing with sync
* 09:03 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2230.codfw.wmnet with reason: host reimage
* 09:03 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2231.codfw.wmnet with reason: host reimage
* 09:01 hashar@deploy2002: anzx, hashar: Backport for gerrit:1111661Add dso and thq to wmgExtraLanguageNames (T383785) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 09:01 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72098 and previous config saved to /var/cache/conftool/dbconfig/20250116-090102-root.json
* 08:58 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2230.codfw.wmnet with reason: host reimage
* 08:55 hashar@deploy2002: Started scap sync-world: Backport for gerrit:1111661Add dso and thq to wmgExtraLanguageNames (T383785)
* 08:55 marostegui@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es1025.eqiad.wmnet with reason: cloning
* 08:54 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1025', diff saved to https://phabricator.wikimedia.org/P72097 and previous config saved to /var/cache/conftool/dbconfig/20250116-085439-marostegui.json
* 08:53 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2229.codfw.wmnet with reason: host reimage
* 08:53 marostegui@cumin1002: dbctl commit (dc=all): 'Promote es1023 to es5 primary T382569', diff saved to https://phabricator.wikimedia.org/P72096 and previous config saved to /var/cache/conftool/dbconfig/20250116-085305-root.json
* 08:51 hashar@deploy2002: Finished scap sync-world: Backport for gerrit:1111912mediawiki.widgets: Fix aborting TitleWidget request breaking it permanently (T383497), gerrit:1111913mediawiki.widgets: Fix aborting TitleWidget request breaking it permanently (T383497) (duration: 14m 53s)
* 08:50 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2229.codfw.wmnet with reason: host reimage
* 08:49 marostegui@cumin1002: dbctl commit (dc=all): 'es1045 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P72095 and previous config saved to /var/cache/conftool/dbconfig/20250116-084946-root.json
* 08:46 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2231
* 08:45 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2231
* 08:45 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72094 and previous config saved to /var/cache/conftool/dbconfig/20250116-084557-root.json
* 08:45 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2231
* 08:45 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2231.codfw.wmnet 207.32.192.10.in-addr.arpa 7.0.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:45 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2228.codfw.wmnet with reason: host reimage
* 08:45 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2231.codfw.wmnet 207.32.192.10.in-addr.arpa 7.0.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:45 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:45 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2231 - jelto@cumin1002"
* 08:45 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2231 - jelto@cumin1002"
* 08:42 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 08:41 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2231
* 08:41 hashar@deploy2002: matmarex, hashar: Continuing with sync
* 08:41 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2231.codfw.wmnet with OS bookworm
* 08:41 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2228.codfw.wmnet with reason: host reimage
* 08:40 hashar@deploy2002: matmarex, hashar: Backport for gerrit:1111912mediawiki.widgets: Fix aborting TitleWidget request breaking it permanently (T383497), gerrit:1111913mediawiki.widgets: Fix aborting TitleWidget request breaking it permanently (T383497) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 08:40 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2230
* 08:40 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2230
* 08:39 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2230
* 08:39 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2230.codfw.wmnet 206.32.192.10.in-addr.arpa 6.0.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:39 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2230.codfw.wmnet 206.32.192.10.in-addr.arpa 6.0.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:39 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:39 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2230 - jelto@cumin1002"
* 08:39 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2230 - jelto@cumin1002"
* 08:36 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 08:36 hashar@deploy2002: Started scap sync-world: Backport for gerrit:1111912mediawiki.widgets: Fix aborting TitleWidget request breaking it permanently (T383497), gerrit:1111913mediawiki.widgets: Fix aborting TitleWidget request breaking it permanently (T383497)
* 08:36 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2230
* 08:36 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2230.codfw.wmnet with OS bookworm
* 08:36 marostegui@cumin1002: dbctl commit (dc=all): 'Add es1045 to dbctl depooled T382569', diff saved to https://phabricator.wikimedia.org/P72093 and previous config saved to /var/cache/conftool/dbconfig/20250116-083559-marostegui.json
* 08:32 oblivian@deploy2002: Finished scap sync-world: Backport for gerrit:1109108ClusterConfig: add support for dumps trait (T382947) (duration: 12m 30s)
* 08:32 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2229
* 08:32 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2229
* 08:32 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2229
* 08:32 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2229.codfw.wmnet 205.32.192.10.in-addr.arpa 5.0.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:32 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2229.codfw.wmnet 205.32.192.10.in-addr.arpa 5.0.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:32 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:32 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2229 - jelto@cumin1002"
* 08:32 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2229 - jelto@cumin1002"
* 08:30 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72092 and previous config saved to /var/cache/conftool/dbconfig/20250116-083051-root.json
* 08:28 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 08:28 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2229
* 08:28 oblivian@deploy2002: oblivian: Continuing with sync
* 08:28 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2229.codfw.wmnet with OS bookworm
* 08:26 oblivian@deploy2002: oblivian: Backport for gerrit:1109108ClusterConfig: add support for dumps trait (T382947) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 08:23 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2228
* 08:23 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2228
* 08:23 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2228
* 08:23 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2228.codfw.wmnet 204.32.192.10.in-addr.arpa 4.0.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:23 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2228.codfw.wmnet 204.32.192.10.in-addr.arpa 4.0.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:23 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:23 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2228 - jelto@cumin1002"
* 08:23 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2228 - jelto@cumin1002"
* 08:20 oblivian@deploy2002: Started scap sync-world: Backport for gerrit:1109108ClusterConfig: add support for dumps trait (T382947)
* 08:20 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 08:19 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2228
* 08:19 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2228.codfw.wmnet with OS bookworm
* 08:17 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2228.codfw.wmnet wikikube-worker2229.codfw.wmnet wikikube-worker2230.codfw.wmnet wikikube-worker2231.codfw.wmnet on all recursors
* 08:17 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2228.codfw.wmnet wikikube-worker2229.codfw.wmnet wikikube-worker2230.codfw.wmnet wikikube-worker2231.codfw.wmnet on all recursors
* 08:17 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2357 to wikikube-worker2231
* 08:16 oblivian@deploy2002: Finished scap sync-world: Backport for gerrit:987432Explicitly disable all local imagescaling on k8s (T352515) (duration: 12m 35s)
* 08:16 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2231
* 08:16 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2231
* 08:16 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:16 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2357 to wikikube-worker2231 - jelto@cumin1002"
* 08:15 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2357 to wikikube-worker2231 - jelto@cumin1002"
* 08:12 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 08:12 oblivian@deploy2002: oblivian: Continuing with sync
* 08:12 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2357 to wikikube-worker2231
* 08:11 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2356 to wikikube-worker2230
* 08:10 oblivian@deploy2002: oblivian: Backport for gerrit:987432Explicitly disable all local imagescaling on k8s (T352515) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 08:10 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2230
* 08:10 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2230
* 08:10 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:10 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2356 to wikikube-worker2230 - jelto@cumin1002"
* 08:10 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2356 to wikikube-worker2230 - jelto@cumin1002"
* 08:06 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 08:06 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2356 to wikikube-worker2230
* 08:05 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2355 to wikikube-worker2229
* 08:04 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2229
* 08:04 oblivian@deploy2002: Started scap sync-world: Backport for gerrit:987432Explicitly disable all local imagescaling on k8s (T352515)
* 08:04 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2229
* 08:04 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:04 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2355 to wikikube-worker2229 - jelto@cumin1002"
* 08:03 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2355 to wikikube-worker2229 - jelto@cumin1002"
* 08:00 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 07:59 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2355 to wikikube-worker2229
* 07:59 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2354 to wikikube-worker2228
* 07:58 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2228
* 07:58 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2228
* 07:58 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:58 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2354 to wikikube-worker2228 - jelto@cumin1002"
* 07:57 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2354 to wikikube-worker2228 - jelto@cumin1002"
* 07:54 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 07:53 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2354 to wikikube-worker2228
* 07:47 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[2354-2357].codfw.wmnet
* 07:45 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[2354-2357].codfw.wmnet
* 07:02 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 07:02 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 03:21 eileen: civicrm upgraded from to 
* 01:01 eileen: config revision changed from to 
* 00:21 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T370903)', diff saved to https://phabricator.wikimedia.org/P72091 and previous config saved to /var/cache/conftool/dbconfig/20250116-002137-ladsgroup.json
* 00:13 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0)
* 00:06 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P72090 and previous config saved to /var/cache/conftool/dbconfig/20250116-000630-ladsgroup.json
== 2025-01-15 ==
* 23:52 ryankemper@cumin2002: START - Cookbook sre.wdqs.restart
* 23:51 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P72089 and previous config saved to /var/cache/conftool/dbconfig/20250115-235123-ladsgroup.json
* 23:38 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0)
* 23:36 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T370903)', diff saved to https://phabricator.wikimedia.org/P72088 and previous config saved to /var/cache/conftool/dbconfig/20250115-233617-ladsgroup.json
* 23:27 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1163 (T370903)', diff saved to https://phabricator.wikimedia.org/P72087 and previous config saved to /var/cache/conftool/dbconfig/20250115-232737-ladsgroup.json
* 23:27 ladsgroup@cumin1002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1163.eqiad.wmnet with reason: Maintenance
* 22:49 cwhite: restarted thanos-query-fronted on titan100[12]
* 22:45 ryankemper@cumin2002: START - Cookbook sre.wdqs.restart
* 22:41 ryankemper@cumin2002: END (ERROR) - Cookbook sre.wdqs.restart (exit_code=97)
* 22:36 ryankemper@cumin2002: START - Cookbook sre.wdqs.restart
* 22:35 ryankemper@cumin2002: END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99)
* 22:35 ryankemper@cumin2002: START - Cookbook sre.wdqs.restart
* 22:18 dmartin@deploy2002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 22:17 dmartin@deploy2002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 22:17 dmartin@deploy2002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 22:15 dmartin@deploy2002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 22:12 dmartin@deploy2002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 22:10 dmartin@deploy2002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 21:40 tzatziki: Removing 10 files for legal compliance
* 21:22 samtar@deploy2002: Finished scap sync-world: Backport for gerrit:1111706do not allow temp users to edit on loginwiki (T383747) (duration: 09m 34s)
* 21:19 otto@deploy2002: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: sync
* 21:19 otto@deploy2002: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: sync
* 21:18 otto@deploy2002: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: sync
* 21:18 samtar@deploy2002: samtar, jjmc89: Continuing with sync
* 21:17 samtar@deploy2002: samtar, jjmc89: Backport for gerrit:1111706do not allow temp users to edit on loginwiki (T383747) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:17 otto@deploy2002: helmfile [codfw] START helmfile.d/services/eventgate-analytics: sync
* 21:17 tzatziki: Removing 11 files for legal compliance
* 21:16 otto@deploy2002: helmfile [staging] DONE helmfile.d/services/eventgate-analytics: sync
* 21:16 otto@deploy2002: helmfile [staging] START helmfile.d/services/eventgate-analytics: sync
* 21:16 ottomata: roll restarting eventgate-analytics to pick up new stream configuration for haproxy_requestctl
* 21:13 samtar@deploy2002: Started scap sync-world: Backport for gerrit:1111706do not allow temp users to edit on loginwiki (T383747)
* 20:54 tzatziki: Removing 1 file for legal compliance
* 20:31 tzatziki: Removing 1 file for legal compliance
* 20:28 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1012.eqiad.wmnet with OS bookworm
* 20:20 tzatziki: Removing 2 files for legal compliance
* 20:11 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T371742)', diff saved to https://phabricator.wikimedia.org/P72085 and previous config saved to /var/cache/conftool/dbconfig/20250115-201152-ladsgroup.json
* 20:01 brennen@deploy2002: rebuilt and synchronized wikiversions files: group1 to 1.44.0-wmf.12 refs T382363
* 19:59 tzatziki: Removing 1 file for legal compliance
* 19:56 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P72083 and previous config saved to /var/cache/conftool/dbconfig/20250115-195645-ladsgroup.json
* 19:55 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bookworm
* 19:54 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1012.eqiad.wmnet with OS bookworm
* 19:46 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bookworm
* 19:46 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1012.eqiad.wmnet with OS bookworm
* 19:41 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P72082 and previous config saved to /var/cache/conftool/dbconfig/20250115-194138-ladsgroup.json
* 19:39 brennen@deploy2002: Finished scap sync-world: Backport for gerrit:1111680RevisionStore: No first revision of non-existing page (T380677) (duration: 17m 00s)
* 19:34 brennen@deploy2002: bpirkle, brennen: Continuing with sync
* 19:29 brennen@deploy2002: bpirkle, brennen: Backport for gerrit:1111680RevisionStore: No first revision of non-existing page (T380677) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 19:26 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T371742)', diff saved to https://phabricator.wikimedia.org/P72081 and previous config saved to /var/cache/conftool/dbconfig/20250115-192631-ladsgroup.json
* 19:22 brennen@deploy2002: Started scap sync-world: Backport for gerrit:1111680RevisionStore: No first revision of non-existing page (T380677)
* 19:21 aqu@deploy2002: Finished deploy [airflow-dags/analytics@cd03eb7]: Cascading backfill under projectview hourly (duration: 01m 06s)
* 19:20 aqu@deploy2002: Started deploy [airflow-dags/analytics@cd03eb7]: Cascading backfill under projectview hourly
* 19:15 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bookworm
* 19:15 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1012.eqiad.wmnet with OS bookworm
* 19:08 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bookworm
* 19:08 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1012.eqiad.wmnet with OS bookworm
* 19:07 sukhe@cumin1002: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool site ulsfo [reason: this is a test, not actual depool, no task ID specified]
* 19:07 sukhe@cumin1002: START - Cookbook sre.dns.admin DNS admin: depool site ulsfo [reason: this is a test, not actual depool, no task ID specified]
* 18:51 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bookworm
* 18:50 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1012.eqiad.wmnet with OS bookworm
* 18:22 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bookworm
* 18:22 andrew@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudcephosd1012.eqiad.wmnet with OS bookworm
* 18:16 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1163 (T371742)', diff saved to https://phabricator.wikimedia.org/P72080 and previous config saved to /var/cache/conftool/dbconfig/20250115-181629-ladsgroup.json
* 18:16 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1163.eqiad.wmnet with reason: Maintenance
* 18:16 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1163.eqiad.wmnet with reason: Maintenance
* 18:09 volans@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:05:00 on sretest1002.eqiad.wmnet with reason: testing cookbook
* 18:06 volans@cumin2002: START - Cookbook sre.hosts.downtime for 0:05:00 on sretest1002.eqiad.wmnet with reason: testing cookbook
* 18:05 volans@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:05:00 on sretest1001.eqiad.wmnet with reason: testing cookbook
* 17:59 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1111166Added new stream config for haproxy_requestctl (T383392) (duration: 13m 38s)
* 17:58 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bookworm
* 17:56 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cloudcephosd1012.eqiad.wmnet
* 17:54 marostegui@cumin1002: dbctl commit (dc=all): 'db1163 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72079 and previous config saved to /var/cache/conftool/dbconfig/20250115-175456-root.json
* 17:53 cmooney@cumin1002: START - Cookbook sre.hosts.dhcp for host cloudcephosd1012.eqiad.wmnet
* 17:53 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1102-1106].eqiad.wmnet
* 17:53 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1102-1106].eqiad.wmnet
* 17:52 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, fabfur: Continuing with sync
* 17:52 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, fabfur: Backport for gerrit:1111166Added new stream config for haproxy_requestctl (T383392) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 17:52 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1103.eqiad.wmnet with OS bookworm
* 17:51 kamila_: homer cr*eqiad* commit T377876
* 17:49 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1102.eqiad.wmnet with OS bookworm
* 17:47 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1105.eqiad.wmnet with OS bookworm
* 17:45 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1111166Added new stream config for haproxy_requestctl (T383392)
* 17:43 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1106.eqiad.wmnet with OS bookworm
* 17:39 marostegui@cumin1002: dbctl commit (dc=all): 'db1163 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72077 and previous config saved to /var/cache/conftool/dbconfig/20250115-173951-root.json
* 17:36 andrew@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1012.eqiad.wmnet with OS bookworm
* 17:34 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1103.eqiad.wmnet with reason: host reimage
* 17:30 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1102.eqiad.wmnet with reason: host reimage
* 17:26 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1105.eqiad.wmnet with reason: host reimage
* 17:26 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bookworm
* 17:26 andrew@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1012.eqiad.wmnet with OS bookworm
* 17:25 root@cumin1002: END (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for dbprov1006.eqiad.wmnet: Renew puppet certificate - root@cumin1002
* 17:24 marostegui@cumin1002: dbctl commit (dc=all): 'db1163 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72076 and previous config saved to /var/cache/conftool/dbconfig/20250115-172445-root.json
* 17:24 hnowlan: running `decommission` for 5 codfw jobrunners
* 17:24 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1106.eqiad.wmnet with reason: host reimage
* 17:22 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1103.eqiad.wmnet with reason: host reimage
* 17:21 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1105.eqiad.wmnet with reason: host reimage
* 17:21 root@cumin1002: START - Cookbook sre.puppet.renew-cert for dbprov1006.eqiad.wmnet: Renew puppet certificate - root@cumin1002
* 17:21 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1102.eqiad.wmnet with reason: host reimage
* 17:21 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1106.eqiad.wmnet with reason: host reimage
* 17:18 eevans@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:cassandra-dev: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002
* 17:09 marostegui@cumin1002: dbctl commit (dc=all): 'db1163 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72075 and previous config saved to /var/cache/conftool/dbconfig/20250115-170940-root.json
* 17:06 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1103
* 17:06 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1103
* 17:06 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1103.eqiad.wmnet with OS bookworm
* 17:05 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bookworm
* 17:05 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1102
* 17:05 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1102
* 17:05 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1102.eqiad.wmnet with OS bookworm
* 17:05 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1105
* 17:05 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1105
* 17:05 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1106
* 17:05 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1106
* 17:05 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1106.eqiad.wmnet with OS bookworm
* 17:05 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1105.eqiad.wmnet with OS bookworm
* 17:04 andrew@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1012.eqiad.wmnet with OS bookworm
* 17:02 kamila@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker1102.eqiad.wmnet with OS bookworm
* 17:02 kamila@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker1106.eqiad.wmnet with OS bookworm
* 17:02 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1105.eqiad.wmnet with OS bookworm
* 17:01 kamila@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker1103.eqiad.wmnet with OS bookworm
* 16:59 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1104.eqiad.wmnet with OS bookworm
* 16:58 eevans@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching A:cassandra-dev: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002
* 16:56 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bookworm
* 16:54 marostegui@cumin1002: dbctl commit (dc=all): 'db1163 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72074 and previous config saved to /var/cache/conftool/dbconfig/20250115-165434-root.json
* 16:53 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1106
* 16:53 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1106
* 16:53 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1106.eqiad.wmnet with OS bookworm
* 16:53 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1103
* 16:53 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1103
* 16:53 kamila@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker1106.eqiad.wmnet with OS bookworm
* 16:53 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1103.eqiad.wmnet with OS bookworm
* 16:53 kamila@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker1103.eqiad.wmnet with OS bookworm
* 16:53 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1102
* 16:52 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1102
* 16:52 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1102.eqiad.wmnet with OS bookworm
* 16:52 kamila@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker1102.eqiad.wmnet with OS bookworm
* 16:43 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1105.eqiad.wmnet with reason: host reimage
* 16:40 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1104.eqiad.wmnet with reason: host reimage
* 16:36 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1105.eqiad.wmnet with reason: host reimage
* 16:36 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1104.eqiad.wmnet with reason: host reimage
* 16:21 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1106
* 16:21 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1106
* 16:21 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1106.eqiad.wmnet with OS bookworm
* 16:21 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1105
* 16:21 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1105
* 16:21 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1105.eqiad.wmnet with OS bookworm
* 16:21 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1104
* 16:21 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1104
* 16:20 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1104.eqiad.wmnet with OS bookworm
* 16:20 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1103
* 16:20 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1103
* 16:20 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1103.eqiad.wmnet with OS bookworm
* 16:20 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1102
* 16:20 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1102
* 16:20 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1102.eqiad.wmnet with OS bookworm
* 16:20 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2192.codfw.wmnet
* 16:20 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2192.codfw.wmnet
* 16:17 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1102.eqiad.wmnet wikikube-worker1103.eqiad.wmnet wikikube-worker1104.eqiad.wmnet wikikube-worker1105.eqiad.wmnet wikikube-worker1106.eqiad.wmnet on all recursors
* 16:17 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1102.eqiad.wmnet wikikube-worker1103.eqiad.wmnet wikikube-worker1104.eqiad.wmnet wikikube-worker1105.eqiad.wmnet wikikube-worker1106.eqiad.wmnet on all recursors
* 16:17 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1425 to wikikube-worker1106
* 16:17 jelto: homer 'lsw1-d8-codfw*' commit 'T377877'
* 16:16 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1106
* 16:16 volans@cumin2002: END (ERROR) - Cookbook sre.dns.netbox (exit_code=97)
* 16:16 volans@cumin2002: START - Cookbook sre.dns.netbox
* 16:15 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2192.codfw.wmnet with OS bookworm
* 16:15 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jelto@cumin1002"
* 16:14 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jelto@cumin1002"
* 16:14 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1106
* 16:14 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:14 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1425 to wikikube-worker1106 - kamila@cumin1002"
* 16:13 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1425 to wikikube-worker1106 - kamila@cumin1002"
* 16:09 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 16:09 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1424 to wikikube-worker1105
* 16:08 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1425 to wikikube-worker1106
* 16:08 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1105
* 16:07 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1105
* 16:07 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:07 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1424 to wikikube-worker1105 - kamila@cumin1002"
* 16:06 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1424 to wikikube-worker1105 - kamila@cumin1002"
* 16:04 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1423 to wikikube-worker1104
* 16:04 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1104
* 16:02 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 16:02 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1104
* 16:02 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:02 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1423 to wikikube-worker1104 - kamila@cumin1002"
* 16:02 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1423 to wikikube-worker1104 - kamila@cumin1002"
* 16:01 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1424 to wikikube-worker1105
* 16:00 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd2004-dev.codfw.wmnet with OS bullseye
* 16:00 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1422 to wikikube-worker1103
* 16:00 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov1006.eqiad.wmnet with reason: os upgrade
* 15:59 jynus@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov1006.eqiad.wmnet with reason: os upgrade
* 15:59 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1103
* 15:58 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 15:58 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1103
* 15:58 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:58 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1422 to wikikube-worker1103 - kamila@cumin1002"
* 15:58 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1422 to wikikube-worker1103 - kamila@cumin1002"
* 15:57 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1423 to wikikube-worker1104
* 15:54 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 15:54 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2130.codfw.wmnet
* 15:54 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:54 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2192.codfw.wmnet with reason: host reimage
* 15:53 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1421 to wikikube-worker1102
* 15:53 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1102
* 15:53 moritzm: installing libsoup2.4 security updates
* 15:52 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1422 to wikikube-worker1103
* 15:51 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 15:51 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1102
* 15:51 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:51 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1421 to wikikube-worker1102 - kamila@cumin1002"
* 15:51 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1421 to wikikube-worker1102 - kamila@cumin1002"
* 15:50 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2192.codfw.wmnet with reason: host reimage
* 15:48 root@cumin1002: END (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for dbprov1005.eqiad.wmnet: Renew puppet certificate - root@cumin1002
* 15:47 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2130.codfw.wmnet
* 15:46 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 15:46 root@cumin1002: START - Cookbook sre.puppet.renew-cert for dbprov1005.eqiad.wmnet: Renew puppet certificate - root@cumin1002
* 15:46 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1421 to wikikube-worker1102
* 15:46 root@cumin1002: END (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for dbprov1004.eqiad.wmnet: Renew puppet certificate - root@cumin1002
* 15:43 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[1421-1425].eqiad.wmnet
* 15:43 root@cumin1002: START - Cookbook sre.puppet.renew-cert for dbprov1004.eqiad.wmnet: Renew puppet certificate - root@cumin1002
* 15:40 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[1421-1425].eqiad.wmnet
* 15:36 moritzm: installing python-django security updates
* 15:33 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:32 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:30 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2224-2227].codfw.wmnet
* 15:30 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2224-2227].codfw.wmnet
* 15:30 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2192
* 15:30 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2192
* 15:30 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2192.codfw.wmnet with OS bookworm
* 15:28 volans: uploaded spicerack_9.1.0 to apt.wikimedia.org bullseye-wikimedia
* 15:26 jelto: homer 'cr*codfw*' commit 'T377877'
* 15:25 jelto: homer 'lsw1-c6-codfw*' commit 'T377877'
* 15:23 jelto: homer 'lsw1-d3-codfw*' commit 'T377877'
* 15:22 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2225.codfw.wmnet with OS bookworm
* away: UTC afternoon deploys done
* 15:15 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2226.codfw.wmnet with OS bookworm
* 15:13 tgr@deploy2002: Finished scap sync-world: Backport for gerrit:1111585Enable SUL3 on test wikis, second attempt (T383729) (duration: 27m 56s)
* 15:13 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov1005.eqiad.wmnet with reason: os upgrade
* 15:12 jynus@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov1005.eqiad.wmnet with reason: os upgrade
* 15:12 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2224.codfw.wmnet with OS bookworm
* 15:09 tgr@deploy2002: tgr: Continuing with sync
* 15:02 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2225.codfw.wmnet with reason: host reimage
* 14:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2226.codfw.wmnet with reason: host reimage
* 14:52 tgr@deploy2002: tgr: Backport for gerrit:1111585Enable SUL3 on test wikis, second attempt (T383729) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:51 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2224.codfw.wmnet with reason: host reimage
* 14:48 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2225.codfw.wmnet with reason: host reimage
* 14:48 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2226.codfw.wmnet with reason: host reimage
* 14:48 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2224.codfw.wmnet with reason: host reimage
* 14:45 tgr@deploy2002: Started scap sync-world: Backport for gerrit:1111585Enable SUL3 on test wikis, second attempt (T383729)
* 14:43 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1111343Yet more authentication domain overrides (T383729 T373737), gerrit:1111598Move Beta Cluster favicons to this repository, gerrit:1075211Replace favicon.php with static.php (T374997) (duration: 15m 52s)
* 14:41 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2227.codfw.wmnet with OS bookworm
* 14:40 aqu: Deployed refinery using scap, then deployed onto hdfs
* 14:38 lucaswerkmeister-wmde@deploy2002: tgr, matmarex, lucaswerkmeister-wmde: Continuing with sync
* 14:36 aqu@deploy2002: Finished deploy [analytics/refinery@eee2443] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@eee24435] (duration: 00m 28s)
* 14:35 aqu@deploy2002: Started deploy [analytics/refinery@eee2443] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@eee24435]
* 14:35 aqu@deploy2002: Finished deploy [analytics/refinery@eee2443] (thin): Regular analytics weekly train THIN [analytics/refinery@eee24435] (duration: 00m 33s)
* 14:35 lucaswerkmeister-wmde@deploy2002: tgr, matmarex, lucaswerkmeister-wmde: Backport for gerrit:1111343Yet more authentication domain overrides (T383729 T373737), gerrit:1111598Move Beta Cluster favicons to this repository, gerrit:1075211Replace favicon.php with static.php (T374997) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:34 aqu@deploy2002: Started deploy [analytics/refinery@eee2443] (thin): Regular analytics weekly train THIN [analytics/refinery@eee24435]
* 14:34 aqu@deploy2002: Finished deploy [analytics/refinery@eee2443]: Regular analytics weekly train [analytics/refinery@eee24435] (duration: 01m 58s)
* 14:32 aqu@deploy2002: Started deploy [analytics/refinery@eee2443]: Regular analytics weekly train [analytics/refinery@eee24435]
* 14:32 aqu: Deploying analytics/refinery
* 14:31 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2226
* 14:31 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2226
* 14:31 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2226.codfw.wmnet with OS bookworm
* 14:30 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2225
* 14:30 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2225
* 14:30 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2225.codfw.wmnet with OS bookworm
* 14:30 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2224
* 14:30 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2224
* 14:30 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2224.codfw.wmnet with OS bookworm
* 14:30 jelto@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker2226.codfw.wmnet with OS bookworm
* 14:30 jelto@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker2225.codfw.wmnet with OS bookworm
* 14:29 jelto@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker2224.codfw.wmnet with OS bookworm
* 14:27 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1111343Yet more authentication domain overrides (T383729 T373737), gerrit:1111598Move Beta Cluster favicons to this repository, gerrit:1075211Replace favicon.php with static.php (T374997)
* 14:24 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1111344Add entry point names to all entry points under w/ (T383729) (duration: 16m 48s)
* 14:22 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2227.codfw.wmnet with reason: host reimage
* 14:18 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2227.codfw.wmnet with reason: host reimage
* 14:18 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, tgr: Continuing with sync
* 14:17 jiji@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
* 14:16 jiji@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply
* 14:14 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, tgr: Backport for gerrit:1111344Add entry point names to all entry points under w/ (T383729) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:14 dcausse: repooling wdqs1013
* 14:08 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1111344Add entry point names to all entry points under w/ (T383729)
* 14:02 jiji@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
* 14:02 jiji@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply
* 14:00 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2227
* 14:00 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2227
* 14:00 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2227
* 14:00 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2227.codfw.wmnet 18.48.192.10.in-addr.arpa 8.1.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:00 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2227.codfw.wmnet 18.48.192.10.in-addr.arpa 8.1.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:00 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:00 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2227 - jelto@cumin1002"
* 14:00 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2227 - jelto@cumin1002"
* 13:56 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:56 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2227
* 13:56 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2226
* 13:56 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2226
* 13:55 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2226
* 13:55 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2226.codfw.wmnet 17.48.192.10.in-addr.arpa 7.1.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:55 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2226.codfw.wmnet 17.48.192.10.in-addr.arpa 7.1.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:55 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:55 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2226 - jelto@cumin1002"
* 13:55 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2226 - jelto@cumin1002"
* 13:53 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 13:53 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 13:52 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2227.codfw.wmnet with OS bookworm
* 13:51 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:51 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2226
* 13:51 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2225
* 13:51 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2225
* 13:51 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2225
* 13:51 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2225.codfw.wmnet 15.48.192.10.in-addr.arpa 5.1.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:51 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2225.codfw.wmnet 15.48.192.10.in-addr.arpa 5.1.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:51 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:51 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2225 - jelto@cumin1002"
* 13:51 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2225 - jelto@cumin1002"
* 13:48 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2226.codfw.wmnet with OS bookworm
* 13:47 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2224
* 13:47 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2224
* 13:47 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:47 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2224
* 13:47 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2224.codfw.wmnet 209.32.192.10.in-addr.arpa 9.0.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:47 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2224.codfw.wmnet 209.32.192.10.in-addr.arpa 9.0.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:47 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:47 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2224 - jelto@cumin1002"
* 13:47 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2224 - jelto@cumin1002"
* 13:47 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2225
* 13:46 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2225.codfw.wmnet with OS bookworm
* 13:44 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:43 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2224
* 13:43 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2224.codfw.wmnet with OS bookworm
* 13:40 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2224.codfw.wmnet wikikube-worker2225.codfw.wmnet wikikube-worker2226.codfw.wmnet wikikube-worker2227.codfw.wmnet on all recursors
* 13:40 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2224.codfw.wmnet wikikube-worker2225.codfw.wmnet wikikube-worker2226.codfw.wmnet wikikube-worker2227.codfw.wmnet on all recursors
* 13:39 dcausse: errata: depooling wdqs1013 (high lag)
* 13:39 dcausse: depooling wdqs1003 (high lag)
* 13:36 moritzm: imported firmware-amd-graphics 20230210-5~bpo11+1 into component/amd-gpu-firmware for bullseye-wikimedia T383557
* 13:31 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov1004.eqiad.wmnet with reason: os upgrade
* 13:30 jynus@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov1004.eqiad.wmnet with reason: os upgrade
* 13:30 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2368 to wikikube-worker2227
* 13:30 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2227
* 13:29 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2227
* 13:29 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:29 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2368 to wikikube-worker2227 - jelto@cumin1002"
* 13:22 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
* 13:22 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply
* 13:21 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
* 13:21 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
* 13:14 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1163.eqiad.wmnet with reason: maintenance
* 13:14 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1163.eqiad.wmnet with reason: maintenance
* 13:09 elukey: restart wdqs-blazegraph.service on wdqs1013 - failed http probes
* 13:06 elukey: `elukey@puppetmaster1001:~$ sudo puppet cert destroy graphite.discovery.wmnet`
* 13:05 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1163 T383689', diff saved to https://phabricator.wikimedia.org/P72073 and previous config saved to /var/cache/conftool/dbconfig/20250115-130511-marostegui.json
* 13:03 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1184 to s1 primary T383689', diff saved to https://phabricator.wikimedia.org/P72072 and previous config saved to /var/cache/conftool/dbconfig/20250115-130354-root.json
* 13:03 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2368 to wikikube-worker2227 - jelto@cumin1002"
* 13:03 marostegui: Starting s1 eqiad failover from db1163 to db1184 - T383689
* 12:59 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 12:58 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2368 to wikikube-worker2227
* 12:58 marostegui@cumin1002: dbctl commit (dc=all): 'Remove db1184 from API T383689', diff saved to https://phabricator.wikimedia.org/P72071 and previous config saved to /var/cache/conftool/dbconfig/20250115-125818-root.json
* 12:58 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2367 to wikikube-worker2226
* 12:57 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 34 hosts with reason: Primary switchover s1 T383689
* 12:57 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2226
* 12:57 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1184 with weight 0 T383689', diff saved to https://phabricator.wikimedia.org/P72070 and previous config saved to /var/cache/conftool/dbconfig/20250115-125732-root.json
* 12:57 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2226
* 12:57 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:57 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2367 to wikikube-worker2226 - jelto@cumin1002"
* 12:57 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 34 hosts with reason: Primary switchover s1 T383689
* 12:57 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2367 to wikikube-worker2226 - jelto@cumin1002"
* 12:53 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 12:53 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2367 to wikikube-worker2226
* 12:52 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2366 to wikikube-worker2225
* 12:51 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2225
* 12:51 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2225
* 12:51 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:51 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2366 to wikikube-worker2225 - jelto@cumin1002"
* 12:50 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2366 to wikikube-worker2225 - jelto@cumin1002"
* 12:46 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 12:46 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2366 to wikikube-worker2225
* 12:45 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2359 to wikikube-worker2224
* 12:44 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2224
* 12:44 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2224
* 12:44 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:44 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2359 to wikikube-worker2224 - jelto@cumin1002"
* 12:43 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2359 to wikikube-worker2224 - jelto@cumin1002"
* 12:40 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 12:39 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2359 to wikikube-worker2224
* 12:35 marostegui@cumin1002: dbctl commit (dc=all): 'Remove db2130 from dbctl T383766', diff saved to https://phabricator.wikimedia.org/P72069 and previous config saved to /var/cache/conftool/dbconfig/20250115-123501-marostegui.json
* 12:30 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[2359,2366-2368].codfw.wmnet
* 12:28 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[2359,2366-2368].codfw.wmnet
* 12:24 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2130 T374623', diff saved to https://phabricator.wikimedia.org/P72068 and previous config saved to /var/cache/conftool/dbconfig/20250115-122419-marostegui.json
* 12:10 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db[2134,2160,2234].codfw.wmnet with reason: maintenance
* 12:10 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db[2134,2160,2234].codfw.wmnet with reason: maintenance
* 12:02 kart_: Updated cxserver to 2025-01-15-103159-production (T377966)
* 11:59 kartik@deploy2002: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
* 11:59 kartik@deploy2002: helmfile [eqiad] START helmfile.d/services/cxserver: apply
* 11:58 kartik@deploy2002: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
* 11:58 kartik@deploy2002: helmfile [codfw] START helmfile.d/services/cxserver: apply
* 11:56 kartik@deploy2002: helmfile [staging] DONE helmfile.d/services/cxserver: apply
* 11:56 kartik@deploy2002: helmfile [staging] START helmfile.d/services/cxserver: apply
* 11:32 marostegui@cumin1002: dbctl commit (dc=all): 'es1043 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72067 and previous config saved to /var/cache/conftool/dbconfig/20250115-113210-root.json
* 11:17 marostegui@cumin1002: dbctl commit (dc=all): 'es1043 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72066 and previous config saved to /var/cache/conftool/dbconfig/20250115-111704-root.json
* 11:15 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2220-2223].codfw.wmnet
* 11:15 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2220-2223].codfw.wmnet
* 11:02 marostegui@cumin1002: dbctl commit (dc=all): 'es1043 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72064 and previous config saved to /var/cache/conftool/dbconfig/20250115-110159-root.json
* 10:52 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2222.codfw.wmnet with OS bookworm
* 10:46 marostegui@cumin1002: dbctl commit (dc=all): 'es1043 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72063 and previous config saved to /var/cache/conftool/dbconfig/20250115-104654-root.json
* 10:44 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2223.codfw.wmnet with OS bookworm
* 10:42 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2221.codfw.wmnet with OS bookworm
* 10:36 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2220.codfw.wmnet with OS bookworm
* 10:32 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2222.codfw.wmnet with reason: host reimage
* 10:31 marostegui@cumin1002: dbctl commit (dc=all): 'es1043 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72062 and previous config saved to /var/cache/conftool/dbconfig/20250115-103149-root.json
* 10:25 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2223.codfw.wmnet with reason: host reimage
* 10:21 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2221.codfw.wmnet with reason: host reimage
* 10:19 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2223.codfw.wmnet with reason: host reimage
* 10:18 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2222.codfw.wmnet with reason: host reimage
* 10:17 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2221.codfw.wmnet with reason: host reimage
* 10:16 marostegui@cumin1002: dbctl commit (dc=all): 'es1043 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P72061 and previous config saved to /var/cache/conftool/dbconfig/20250115-101643-root.json
* 10:16 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2220.codfw.wmnet with reason: host reimage
* 10:13 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_magru and A:cp
* 10:13 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2220.codfw.wmnet with reason: host reimage
* 10:08 jayme: re-enabling puppet on nodes using k8s::package resource - T341984
* 10:02 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es1024.eqiad.wmnet with reason: cloning
* 10:02 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on es1024.eqiad.wmnet with reason: cloning
* 10:02 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1024 T382569', diff saved to https://phabricator.wikimedia.org/P72060 and previous config saved to /var/cache/conftool/dbconfig/20250115-100228-marostegui.json
* 10:02 marostegui@cumin1002: dbctl commit (dc=all): 'Promote es1025 to eqiad es5 master dbmaint T382569', diff saved to https://phabricator.wikimedia.org/P72059 and previous config saved to /var/cache/conftool/dbconfig/20250115-100207-marostegui.json
* 10:01 marostegui@cumin1002: dbctl commit (dc=all): 'es1043 (re)pooling @ 4%: Repooling', diff saved to https://phabricator.wikimedia.org/P72058 and previous config saved to /var/cache/conftool/dbconfig/20250115-100138-root.json
* 10:01 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2223
* 10:01 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2223
* 10:01 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2223.codfw.wmnet with OS bookworm
* 10:00 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2222
* 10:00 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2222
* 09:59 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2222.codfw.wmnet with OS bookworm
* 09:59 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2221
* 09:59 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2221
* 09:59 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2221.codfw.wmnet with OS bookworm
* 09:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2220
* 09:55 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2220
* 09:54 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2220.codfw.wmnet with OS bookworm
* 09:54 jayme: disabling puppet on 543 nodes using k8s::package resource - T341984
* 09:51 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2220-2223].codfw.wmnet
* 09:51 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2220-2223].codfw.wmnet
* 09:50 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and A:cp
* 09:47 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp
* 09:46 marostegui@cumin1002: dbctl commit (dc=all): 'es1043 (re)pooling @ 3%: Repooling', diff saved to https://phabricator.wikimedia.org/P72057 and previous config saved to /var/cache/conftool/dbconfig/20250115-094632-root.json
* 09:44 jelto@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2223.codfw.wmnet
* 09:38 jelto@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2222.codfw.wmnet
* 09:34 urbanecm@deploy2002: Finished scap sync-world: Backport for gerrit:1109756Add license messages for new Wikinews licenses (T383338), gerrit:1106911Update French wikinews license to CC-BY-SA 4.0 (T381946), gerrit:1111568htmlform: fix defaults for namespace and relative in titlesmultiselect (T383133), gerrit:1111569htmlform: fix defaults for namespace and relative in titlesmultiselect (T383133) (durat
* 09:31 marostegui@cumin1002: dbctl commit (dc=all): 'es1043 (re)pooling @ 2%: Repooling', diff saved to https://phabricator.wikimedia.org/P72056 and previous config saved to /var/cache/conftool/dbconfig/20250115-093127-root.json
* 09:30 jelto@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2221.codfw.wmnet
* 09:21 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp
* 09:20 urbanecm@deploy2002: matmarex, urbanecm, dreamrimmer: Continuing with sync
* 09:19 marostegui@cumin1002: dbctl commit (dc=all): 'es1022 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72055 and previous config saved to /var/cache/conftool/dbconfig/20250115-091943-root.json
* 09:16 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp
* 09:16 marostegui@cumin1002: dbctl commit (dc=all): 'es1043 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P72054 and previous config saved to /var/cache/conftool/dbconfig/20250115-091622-root.json
* 09:15 urbanecm@deploy2002: matmarex, urbanecm, dreamrimmer: Backport for gerrit:1109756Add license messages for new Wikinews licenses (T383338), gerrit:1106911Update French wikinews license to CC-BY-SA 4.0 (T381946), gerrit:1111568htmlform: fix defaults for namespace and relative in titlesmultiselect (T383133), gerrit:1111569htmlform: fix defaults for namespace and relative in titlesmultiselect (T383133)
* 09:15 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2223.codfw.wmnet with OS bullseye
* 09:11 jelto@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker2220.codfw.wmnet
* 09:10 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2222.codfw.wmnet with OS bullseye
* 09:04 marostegui@cumin1002: dbctl commit (dc=all): 'es1022 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72053 and previous config saved to /var/cache/conftool/dbconfig/20250115-090437-root.json
* 09:02 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2221.codfw.wmnet with OS bullseye
* 08:57 jelto: homer lsw1-d3-codfw* commit 'T377877'
* 08:54 jelto: !log homer cr*codfw* commit 'T377877'
* 08:54 urbanecm@deploy2002: Started scap sync-world: Backport for gerrit:1109756Add license messages for new Wikinews licenses (T383338), gerrit:1106911Update French wikinews license to CC-BY-SA 4.0 (T381946), gerrit:1111568htmlform: fix defaults for namespace and relative in titlesmultiselect (T383133), gerrit:1111569htmlform: fix defaults for namespace and relative in titlesmultiselect (T383133)
* 08:53 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2223.codfw.wmnet with reason: host reimage
* 08:53 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2220.codfw.wmnet with OS bullseye
* 08:53 urbanecm@deploy2002: sync-world aborted: Backport for gerrit:1109756Add license messages for new Wikinews licenses (T383338), gerrit:1106911Update French wikinews license to CC-BY-SA 4.0 (T381946) (duration: 05m 25s)
* 08:53 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp
* 08:51 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2223.codfw.wmnet with reason: host reimage
* 08:49 marostegui@cumin1002: dbctl commit (dc=all): 'es1022 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72052 and previous config saved to /var/cache/conftool/dbconfig/20250115-084932-root.json
* 08:49 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2222.codfw.wmnet with reason: host reimage
* 08:48 urbanecm@deploy2002: Started scap sync-world: Backport for gerrit:1109756Add license messages for new Wikinews licenses (T383338), gerrit:1106911Update French wikinews license to CC-BY-SA 4.0 (T381946)
* 08:45 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2222.codfw.wmnet with reason: host reimage
* 08:44 urbanecm@deploy2002: Finished scap sync-world: Backport for gerrit:1109736Enable abusefilter-log-detail for autoconfirmed users on en.wikibooks (T383332), gerrit:1105420Growth: Remove temporary config for clearing link recommendations (T379522) (duration: 16m 09s)
* 08:42 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2221.codfw.wmnet with reason: host reimage
* 08:38 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2221.codfw.wmnet with reason: host reimage
* 08:37 urbanecm@deploy2002: dreamrimmer, urbanecm, migr: Continuing with sync
* 08:35 urbanecm@deploy2002: dreamrimmer, urbanecm, migr: Backport for gerrit:1109736Enable abusefilter-log-detail for autoconfirmed users on en.wikibooks (T383332), gerrit:1105420Growth: Remove temporary config for clearing link recommendations (T379522) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 08:35 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2223
* 08:35 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2223
* 08:35 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2223
* 08:35 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2223.codfw.wmnet 22.48.192.10.in-addr.arpa 2.2.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:35 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2223.codfw.wmnet 22.48.192.10.in-addr.arpa 2.2.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:35 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:35 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2223 - jelto@cumin1002"
* 08:34 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2223 - jelto@cumin1002"
* 08:34 marostegui@cumin1002: dbctl commit (dc=all): 'es1022 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72051 and previous config saved to /var/cache/conftool/dbconfig/20250115-083427-root.json
* 08:33 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2220.codfw.wmnet with reason: host reimage
* 08:31 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 08:31 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2223
* 08:31 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2223.codfw.wmnet with OS bullseye
* 08:31 jelto@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2223.codfw.wmnet
* 08:30 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2220.codfw.wmnet with reason: host reimage
* 08:30 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2222
* 08:30 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2222
* 08:29 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2222
* 08:29 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2222.codfw.wmnet 21.48.192.10.in-addr.arpa 1.2.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:29 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2222.codfw.wmnet 21.48.192.10.in-addr.arpa 1.2.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:29 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:29 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2222 - jelto@cumin1002"
* 08:29 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2222 - jelto@cumin1002"
* 08:28 urbanecm@deploy2002: Started scap sync-world: Backport for gerrit:1109736Enable abusefilter-log-detail for autoconfirmed users on en.wikibooks (T383332), gerrit:1105420Growth: Remove temporary config for clearing link recommendations (T379522)
* 08:26 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 08:25 marostegui@cumin1002: dbctl commit (dc=all): 'Add es1043 to dbctl depooled T382569', diff saved to https://phabricator.wikimedia.org/P72050 and previous config saved to /var/cache/conftool/dbconfig/20250115-082554-marostegui.json
* 08:25 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2222
* 08:25 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2222.codfw.wmnet with OS bullseye
* 08:25 jelto@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2222.codfw.wmnet
* 08:22 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2221
* 08:22 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2221
* 08:22 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2221
* 08:22 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2221.codfw.wmnet 20.48.192.10.in-addr.arpa 0.2.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:22 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2221.codfw.wmnet 20.48.192.10.in-addr.arpa 0.2.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:22 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:22 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2221 - jelto@cumin1002"
* 08:22 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2221 - jelto@cumin1002"
* 08:19 marostegui@cumin1002: dbctl commit (dc=all): 'es1022 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72049 and previous config saved to /var/cache/conftool/dbconfig/20250115-081922-root.json
* 08:19 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 08:18 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2221
* 08:18 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2221.codfw.wmnet with OS bullseye
* 08:18 jelto@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2221.codfw.wmnet
* 08:14 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2220
* 08:14 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2220
* 08:14 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2220
* 08:14 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2220.codfw.wmnet 19.48.192.10.in-addr.arpa 9.1.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:14 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2220.codfw.wmnet 19.48.192.10.in-addr.arpa 9.1.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:14 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:14 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2220 - jelto@cumin1002"
* 08:13 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2220 - jelto@cumin1002"
* 08:10 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 08:10 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2220
* 08:10 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2220.codfw.wmnet with OS bullseye
* 08:10 jelto@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker2220.codfw.wmnet
* 08:08 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2220.codfw.wmnet wikikube-worker2221.codfw.wmnet wikikube-worker2222.codfw.wmnet wikikube-worker2223.codfw.wmnet on all recursors
* 08:08 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2220.codfw.wmnet wikikube-worker2221.codfw.wmnet wikikube-worker2222.codfw.wmnet wikikube-worker2223.codfw.wmnet on all recursors
* 08:00 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2372 to wikikube-worker2223
* 07:59 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2223
* 07:59 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2223
* 07:59 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:59 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2372 to wikikube-worker2223 - jelto@cumin1002"
* 07:58 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2372 to wikikube-worker2223 - jelto@cumin1002"
* 07:55 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 07:55 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2372 to wikikube-worker2223
* 07:51 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2371 to wikikube-worker2222
* 07:50 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2222
* 07:49 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2222
* 07:49 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:49 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2371 to wikikube-worker2222 - jelto@cumin1002"
* 07:43 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2371 to wikikube-worker2222 - jelto@cumin1002"
* 07:39 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 07:39 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2371 to wikikube-worker2222
* 07:38 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2370 to wikikube-worker2221
* 07:37 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2221
* 07:37 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2221
* 07:37 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:37 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2370 to wikikube-worker2221 - jelto@cumin1002"
* 07:36 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2370 to wikikube-worker2221 - jelto@cumin1002"
* 07:30 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 07:29 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2370 to wikikube-worker2221
* 07:06 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2369 to wikikube-worker2220
* 07:05 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2220
* 07:04 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2220
* 07:04 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:04 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2369 to wikikube-worker2220 - jelto@cumin1002"
* 07:04 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2369 to wikikube-worker2220 - jelto@cumin1002"
* 06:59 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 06:58 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2369 to wikikube-worker2220
* 06:52 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[2369-2372].codfw.wmnet
* 06:50 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[2369-2372].codfw.wmnet
* 05:50 kartik@deploy2002: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply
* 05:30 kartik@deploy2002: helmfile [staging] START helmfile.d/services/machinetranslation: apply
* 05:23 kart_: Updated cxserver to 2025-01-13-044601-production (T382294)
* 05:22 kartik@deploy2002: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
* 05:22 kartik@deploy2002: helmfile [eqiad] START helmfile.d/services/cxserver: apply
* 05:20 kartik@deploy2002: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
* 05:20 kartik@deploy2002: helmfile [codfw] START helmfile.d/services/cxserver: apply
* 05:15 kartik@deploy2002: helmfile [staging] DONE helmfile.d/services/cxserver: apply
* 05:14 kartik@deploy2002: helmfile [staging] START helmfile.d/services/cxserver: apply
* 00:09 tzatziki: removing 1 file for legal compliance
== 2025-01-14 ==
* 23:54 tzatziki: removing 2 files for legal compliance
* 23:41 eileen: civicrm upgraded from to 
* 23:14 tzatziki: removing 2 files for legal compliance
* away: UTC late deploys done
* 22:58 tgr@deploy2002: Finished scap sync-world: Backport for gerrit:1111341Revert "Enable SUL3 on test wikis" (duration: 12m 59s)
* 22:50 tgr@deploy2002: tgr, trainbranchbot: Continuing with sync
* 22:49 tzatziki: removing 5 files for legal compliance
* 22:49 tgr@deploy2002: tgr, trainbranchbot: Backport for gerrit:1111341Revert "Enable SUL3 on test wikis" synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 22:45 tgr@deploy2002: Started scap sync-world: Backport for gerrit:1111341Revert "Enable SUL3 on test wikis"
* 22:31 tzatziki: removing 8 files for legal compliance
* 22:24 tgr@deploy2002: Sync cancelled.
* 22:14 tgr@deploy2002: tgr: Backport for gerrit:1111330Enable SUL3 on test wikis (T383729) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 22:07 tgr@deploy2002: Started scap sync-world: Backport for gerrit:1111330Enable SUL3 on test wikis (T383729)
* 22:02 cjming@deploy2002: Finished scap sync-world: Backport for gerrit:1109694Add azwiki to mobile-anon-talk dblist (T383394) (duration: 17m 03s)
* 21:55 cjming@deploy2002: nmw03, cjming: Continuing with sync
* 21:52 cjming@deploy2002: nmw03, cjming: Backport for gerrit:1109694Add azwiki to mobile-anon-talk dblist (T383394) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:48 swfrench-wmf: deployed conftool 4.2.0 fleet-wide as of ~ 20:00 UTC (previously 4.1.0)
* 21:45 cjming@deploy2002: Started scap sync-world: Backport for gerrit:1109694Add azwiki to mobile-anon-talk dblist (T383394)
* 21:43 cjming@deploy2002: Finished scap sync-world: Backport for gerrit:1106739Fix links pointing to m:Help:Export (duration: 13m 00s)
* 21:36 cjming@deploy2002: tacsipacsi, cjming: Continuing with sync
* 21:35 cjming@deploy2002: tacsipacsi, cjming: Backport for gerrit:1106739Fix links pointing to m:Help:Export synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:30 cjming@deploy2002: Started scap sync-world: Backport for gerrit:1106739Fix links pointing to m:Help:Export
* 21:28 cjming@deploy2002: Finished scap sync-world: Backport for gerrit:1099338SUL3: Add auth domain to URL tests (T380574) (duration: 18m 00s)
* 21:19 cjming@deploy2002: cjming, tgr: Continuing with sync
* 21:17 cjming@deploy2002: cjming, tgr: Backport for gerrit:1099338SUL3: Add auth domain to URL tests (T380574) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:10 cjming@deploy2002: Started scap sync-world: Backport for gerrit:1099338SUL3: Add auth domain to URL tests (T380574)
* 21:05 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1098-1101].eqiad.wmnet
* 21:05 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1098-1101].eqiad.wmnet
* 21:04 bking@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudelastic[1005-1006].eqiad.wmnet
* 21:04 bking@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 21:04 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudelastic[1005-1006].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - bking@cumin2002"
* 21:03 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudelastic[1005-1006].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - bking@cumin2002"
* 20:56 bking@cumin2002: START - Cookbook sre.dns.netbox
* 20:48 bking@cumin2002: START - Cookbook sre.hosts.decommission for hosts cloudelastic[1005-1006].eqiad.wmnet
* 20:32 bking@cumin2002: conftool action : set/pooled=yes:weight=10; selector: name=cloudelastic1012.eqiad.wmnet
* 20:32 bking@cumin2002: conftool action : set/pooled=yes:weight=10; selector: name=cloudelastic1011.eqiad.wmnet
* 20:15 cjming@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
* 20:15 cjming@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
* 20:15 cjming@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
* 20:14 cjming@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
* 20:01 cdanis@cumin2002: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "[not really into teleological thinking] - cdanis@cumin2002"
* 20:01 cdanis@cumin2002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: [not really into teleological thinking] - cdanis@cumin2002
* 20:01 cdanis@cumin2002: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: [not really into teleological thinking] - cdanis@cumin2002
* 20:01 cdanis@cumin2002: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "[not really into teleological thinking] - cdanis@cumin2002"
* 19:42 moritzm: installing rsync security updates on bullseye
* 19:29 brennen@deploy2002: rebuilt and synchronized wikiversions files: group0 to 1.44.0-wmf.12 refs T382363
* 19:12 brennen: 1.44.0-wmf.12 train (T382363): no current blockers, rolling to group0
* 18:57 swfrench@deploy2002: Finished scap sync-world: k8s-only deploy to clear noop chart version diffs (duration: 02m 15s)
* 18:55 swfrench@deploy2002: Started scap sync-world: k8s-only deploy to clear noop chart version diffs
* 18:49 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply
* 18:49 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply
* 18:41 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
* 18:41 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
* 18:26 moritzm: installing rsync security updates on bookworm
* 18:22 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1101.eqiad.wmnet with OS bookworm
* 18:06 kamila@deploy2002: Finished scap sync-world: enable auth.wikimedia.org (duration: 17m 55s)
* 18:04 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1101.eqiad.wmnet with reason: host reimage
* 18:04 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1100.eqiad.wmnet with OS bookworm
* 18:00 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1101.eqiad.wmnet with reason: host reimage
* 17:51 kamila@deploy2002: Started scap sync-world: enable auth.wikimedia.org
* 17:46 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1100.eqiad.wmnet with reason: host reimage
* 17:44 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1101
* 17:44 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1101
* 17:44 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1101.eqiad.wmnet with OS bookworm
* 17:42 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1100.eqiad.wmnet with reason: host reimage
* 17:38 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
* 17:38 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply
* 17:36 swfrench-wmf: continuing shellbox-syntaxhighlight pilot on PHP 8.1 (all eqiad replicas) - T377038
* 17:36 kamila@dns1004: END - running authdns-update
* 17:34 kamila@dns1004: START - running authdns-update
* 17:27 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1100
* 17:27 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1100
* 17:27 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1100.eqiad.wmnet with OS bookworm
* 17:16 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
* 17:15 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply
* 17:15 swfrench-wmf: starting shellbox-syntaxhighlight pilot on PHP 8.1 (1 replica, eqiad only) - T377038
* 17:13 kamila@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1100.eqiad.wmnet with OS bookworm
* 17:13 kamila@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1101.eqiad.wmnet with OS bookworm
* 17:04 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 17:03 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 16:57 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1098.eqiad.wmnet with OS bookworm
* 16:53 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1099.eqiad.wmnet with OS bookworm
* 16:39 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1098.eqiad.wmnet with reason: host reimage
* 16:35 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1099.eqiad.wmnet with reason: host reimage
* 16:32 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1098.eqiad.wmnet with reason: host reimage
* 16:31 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1099.eqiad.wmnet with reason: host reimage
* 16:16 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1098
* 16:16 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1098
* 16:16 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1098.eqiad.wmnet with OS bookworm
* 16:16 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1101
* 16:16 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1101
* 16:16 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1101.eqiad.wmnet with OS bookworm
* 16:16 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1100
* 16:16 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1100
* 16:16 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1100.eqiad.wmnet with OS bookworm
* 16:15 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1099
* 16:15 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1099
* 16:15 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1099.eqiad.wmnet with OS bookworm
* 16:14 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1098.eqiad.wmnet wikikube-worker1099.eqiad.wmnet wikikube-worker1100.eqiad.wmnet wikikube-worker1101.eqiad.wmnet on all recursors
* 16:14 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1098.eqiad.wmnet wikikube-worker1099.eqiad.wmnet wikikube-worker1100.eqiad.wmnet wikikube-worker1101.eqiad.wmnet on all recursors
* 16:13 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1419 to wikikube-worker1101
* 16:13 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1101
* 16:12 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1101
* 16:12 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:11 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1416 to wikikube-worker1100
* 16:10 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1100
* 16:09 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 16:09 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1100
* 16:09 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:07 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1414 to wikikube-worker1098
* 16:07 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1098
* 16:06 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 16:05 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1098
* 16:05 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:05 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1414 to wikikube-worker1098 - kamila@cumin1002"
* 16:05 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1414 to wikikube-worker1098 - kamila@cumin1002"
* 16:03 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1415 to wikikube-worker1099
* 16:02 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1099
* 16:01 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 16:01 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1099
* 16:01 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:01 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1415 to wikikube-worker1099 - kamila@cumin1002"
* 16:00 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1415 to wikikube-worker1099 - kamila@cumin1002"
* 15:58 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1414 to wikikube-worker1098
* 15:56 kamila@cumin1002: END (FAIL) - Cookbook sre.hosts.rename (exit_code=99) from mw1414 to wikikube-worker1098
* 15:56 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 15:56 kamila@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 15:56 kamila@cumin1002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1414 to wikikube-worker1098 - kamila@cumin1002"
* 15:56 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1414 to wikikube-worker1098 - kamila@cumin1002"
* 15:55 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1419 to wikikube-worker1101
* 15:54 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1416 to wikikube-worker1100
* 15:54 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1415 to wikikube-worker1099
* 15:53 moritzm: import prometheus-mysqld-exporter 0.13.0-1~bpo11+1 to the main component of bullseye-wikimedia (import from bullseye-backports which is going away) T383557
* 15:52 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 15:52 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1414 to wikikube-worker1098
* 15:50 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[1414-1416,1419].eqiad.wmnet
* 15:48 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[1414-1416,1419].eqiad.wmnet
* 15:47 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2216-2219].codfw.wmnet
* 15:47 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2216-2219].codfw.wmnet
* 15:47 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
* 15:46 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
* 15:45 jelto: homer 'cr*codfw*' commit 'T377877'
* 15:44 jelto: homer 'lsw1-d3-codfw*' commit 'T377877'
* 15:34 Lucas_WMDE: UTC afternoon backport+config window done
* 15:33 Lucas_WMDE: previous deployment also included gerrit:1111196fix(tracking): TimingMetric:observe records milliseconds (T383208)
* 15:31 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1111253Revert "Make WikibaseQualityConstraints use split-graph query service" (T374021), gerrit:1105878Make WikimediaCampaignEvents use split-graph query service (T377956) (duration: 16m 03s)
* 15:27 marostegui: Stop in sync db2133 db2233 m2 codfw dbmaint T373579
* 15:26 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db[2133,2160,2233].codfw.wmnet with reason: cloning
* 15:26 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 10:00:00 on db[2133,2160,2233].codfw.wmnet with reason: cloning
* 15:24 lucaswerkmeister-wmde@deploy2002: stevemunene, lucaswerkmeister-wmde: Continuing with sync
* 15:21 lucaswerkmeister-wmde@deploy2002: stevemunene, lucaswerkmeister-wmde: Backport for gerrit:1111253Revert "Make WikibaseQualityConstraints use split-graph query service" (T374021), gerrit:1105878Make WikimediaCampaignEvents use split-graph query service (T377956) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 15:15 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1111253Revert "Make WikibaseQualityConstraints use split-graph query service" (T374021), gerrit:1105878Make WikimediaCampaignEvents use split-graph query service (T377956)
* 15:12 lucaswerkmeister-wmde@deploy2002: Sync cancelled.
* 15:04 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
* 15:04 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
* 15:01 lucaswerkmeister-wmde@deploy2002: stevemunene, lucaswerkmeister-wmde: Backport for gerrit:1105879Make WikibaseQualityConstraints use split-graph query service (T374021), gerrit:1105878Make WikimediaCampaignEvents use split-graph query service (T377956) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:59 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2219.codfw.wmnet with OS bookworm
* 14:58 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
* 14:55 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1105879Make WikibaseQualityConstraints use split-graph query service (T374021), gerrit:1105878Make WikimediaCampaignEvents use split-graph query service (T377956)
* 14:53 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1111227test(2)wiki: Explicitly assign event organizer rights to all users (T376822) (duration: 22m 26s)
* 14:47 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_eqsin and A:cp
* 14:46 lucaswerkmeister-wmde@deploy2002: daimona, lucaswerkmeister-wmde: Continuing with sync
* 14:41 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
* 14:39 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2219.codfw.wmnet with reason: host reimage
* 14:37 lucaswerkmeister-wmde@deploy2002: daimona, lucaswerkmeister-wmde: Backport for gerrit:1111227test(2)wiki: Explicitly assign event organizer rights to all users (T376822) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:35 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2219.codfw.wmnet with reason: host reimage
* 14:31 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1111227test(2)wiki: Explicitly assign event organizer rights to all users (T376822)
* 14:29 root@cumin1002: END (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for dbprov1003.eqiad.wmnet: Renew puppet certificate - root@cumin1002
* 14:29 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1109842Enable CampaignEvents extension on idwiki, itwiki, mswiki, and plwiki (T383154) (duration: 23m 59s)
* 14:28 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2218.codfw.wmnet with OS bookworm
* 14:27 root@cumin1002: START - Cookbook sre.puppet.renew-cert for dbprov1003.eqiad.wmnet: Renew puppet certificate - root@cumin1002
* 14:26 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2217.codfw.wmnet with OS bookworm
* 14:22 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2216.codfw.wmnet with OS bookworm
* 14:21 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, daimona: Continuing with sync
* 14:18 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2219
* 14:18 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2219
* 14:18 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2219.codfw.wmnet with OS bookworm
* 14:17 jelto@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker2219.codfw.wmnet with OS bookworm
* 14:15 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, daimona: Backport for gerrit:1109842Enable CampaignEvents extension on idwiki, itwiki, mswiki, and plwiki (T383154) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:09 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2218.codfw.wmnet with reason: host reimage
* 14:05 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2217.codfw.wmnet with reason: host reimage
* 14:05 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2218.codfw.wmnet with reason: host reimage
* 14:05 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1109842Enable CampaignEvents extension on idwiki, itwiki, mswiki, and plwiki (T383154)
* 14:02 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2216.codfw.wmnet with reason: host reimage
* 14:02 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqsin and A:cp
* 14:01 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
* 14:01 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
* 14:01 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2217.codfw.wmnet with reason: host reimage
* 13:57 jayme: imported kubernetes 1.31.4-1 to bookworm-wikimedia - T341984
* 13:56 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2216.codfw.wmnet with reason: host reimage
* 13:53 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2219
* 13:53 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2219
* 13:53 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2219
* 13:53 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2219.codfw.wmnet 148.48.192.10.in-addr.arpa 8.4.1.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:53 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2219.codfw.wmnet 148.48.192.10.in-addr.arpa 8.4.1.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:53 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:53 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2219 - jelto@cumin1002"
* 13:53 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2219 - jelto@cumin1002"
* 13:50 jayme: imported calico 3.29.1-1 to bookworm-wikimedia - T341984
* 13:48 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:48 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2219
* 13:48 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2219.codfw.wmnet with OS bookworm
* 13:48 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2218
* 13:48 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2218
* 13:48 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2218
* 13:48 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2218.codfw.wmnet 147.48.192.10.in-addr.arpa 7.4.1.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:48 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2218.codfw.wmnet 147.48.192.10.in-addr.arpa 7.4.1.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:48 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:48 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2218 - jelto@cumin1002"
* 13:47 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2218 - jelto@cumin1002"
* 13:44 jayme: imported kubernetes 1.23.14-5 to bullseye/bookworm-wikimedia - T341984
* 13:44 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:44 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 13:44 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2218
* 13:44 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 13:44 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2217
* 13:44 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2217
* 13:44 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2217
* 13:43 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2217.codfw.wmnet 146.48.192.10.in-addr.arpa 6.4.1.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:43 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2217.codfw.wmnet 146.48.192.10.in-addr.arpa 6.4.1.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:43 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:43 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2217 - jelto@cumin1002"
* 13:43 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2217 - jelto@cumin1002"
* 13:42 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2218.codfw.wmnet with OS bookworm
* 13:40 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:40 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2217
* 13:39 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2217.codfw.wmnet with OS bookworm
* 13:39 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2216
* 13:39 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2216
* 13:37 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2216
* 13:37 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2216.codfw.wmnet 145.48.192.10.in-addr.arpa 5.4.1.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:37 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2216.codfw.wmnet 145.48.192.10.in-addr.arpa 5.4.1.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:37 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:37 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2216 - jelto@cumin1002"
* 13:37 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2216 - jelto@cumin1002"
* 13:34 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:32 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2216
* 13:32 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2216.codfw.wmnet with OS bookworm
* 13:29 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2216.codfw.wmnet wikikube-worker2217.codfw.wmnet wikikube-worker2218.codfw.wmnet wikikube-worker2219.codfw.wmnet on all recursors
* 13:29 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2216.codfw.wmnet wikikube-worker2217.codfw.wmnet wikikube-worker2218.codfw.wmnet wikikube-worker2219.codfw.wmnet on all recursors
* 13:29 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2376 to wikikube-worker2219
* 13:28 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2219
* 13:28 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2219
* 13:28 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:28 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2376 to wikikube-worker2219 - jelto@cumin1002"
* 13:28 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2376 to wikikube-worker2219 - jelto@cumin1002"
* 13:24 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:24 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2376 to wikikube-worker2219
* 13:23 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2375 to wikikube-worker2218
* 13:22 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2218
* 13:22 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2218
* 13:22 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:22 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2375 to wikikube-worker2218 - jelto@cumin1002"
* 13:22 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2375 to wikikube-worker2218 - jelto@cumin1002"
* 13:17 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:17 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2375 to wikikube-worker2218
* 13:16 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2374 to wikikube-worker2217
* 13:16 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2217
* 13:16 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2217
* 13:16 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:15 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2374 to wikikube-worker2217 - jelto@cumin1002"
* 13:15 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2374 to wikikube-worker2217 - jelto@cumin1002"
* 13:11 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:11 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2374 to wikikube-worker2217
* 13:11 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2373 to wikikube-worker2216
* 13:10 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2216
* 13:10 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2216
* 13:10 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:10 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2373 to wikikube-worker2216 - jelto@cumin1002"
* 13:09 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2373 to wikikube-worker2216 - jelto@cumin1002"
* 13:06 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:06 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2373 to wikikube-worker2216
* 13:01 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[2373-2376].codfw.wmnet
* 12:59 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[2373-2376].codfw.wmnet
* 12:52 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2128.codfw.wmnet
* 12:52 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:52 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2128.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 12:52 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2128.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 12:49 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 12:44 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2128.codfw.wmnet
* 12:41 ladsgroup@cumin1002: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db2212 gradually with 4 steps - Maint over
* 12:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repool pc5 T383398', diff saved to https://phabricator.wikimedia.org/P72044 and previous config saved to /var/cache/conftool/dbconfig/20250114-120804-marostegui.json
* 12:02 marostegui@cumin1002: dbctl commit (dc=all): 'Depool pc5 eqiad codfw dbmaint T383398', diff saved to https://phabricator.wikimedia.org/P72043 and previous config saved to /var/cache/conftool/dbconfig/20250114-120234-marostegui.json
* 12:02 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc[2015,2017].codfw.wmnet,pc[1014-1015,1017].eqiad.wmnet with reason: maintenance
* 12:01 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on pc[2015,2017].codfw.wmnet,pc[1014-1015,1017].eqiad.wmnet with reason: maintenance
* 11:55 ladsgroup@cumin1002: START - Cookbook sre.mysql.pool db2212 gradually with 4 steps - Maint over
* 11:54 vgutierrez@dns1004: END - running authdns-update
* 11:52 vgutierrez@dns1004: START - running authdns-update
* 11:48 marostegui@cumin1002: dbctl commit (dc=all): 'es1044 (re)pooling @ 100%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72041 and previous config saved to /var/cache/conftool/dbconfig/20250114-114804-root.json
* 11:38 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es1020.eqiad.wmnet
* 11:38 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:38 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es1020.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 11:37 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es1020.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 11:34 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 11:33 marostegui@cumin1002: dbctl commit (dc=all): 'es1044 (re)pooling @ 75%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72040 and previous config saved to /var/cache/conftool/dbconfig/20250114-113259-root.json
* 11:28 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts es1020.eqiad.wmnet
* 11:17 marostegui@cumin1002: dbctl commit (dc=all): 'es1044 (re)pooling @ 50%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72039 and previous config saved to /var/cache/conftool/dbconfig/20250114-111754-root.json
* 11:16 marostegui@cumin1002: dbctl commit (dc=all): 'Remove es1020 from dbctl for decommission T383578', diff saved to https://phabricator.wikimedia.org/P72038 and previous config saved to /var/cache/conftool/dbconfig/20250114-111647-marostegui.json
* 11:14 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2239.codfw.wmnet with reason: reboot
* 11:14 jynus@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2239.codfw.wmnet with reason: reboot
* 11:13 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov1003.eqiad.wmnet with reason: os upgrade
* 11:13 jynus@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov1003.eqiad.wmnet with reason: os upgrade
* 11:05 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp
* 11:02 marostegui@cumin1002: dbctl commit (dc=all): 'es1044 (re)pooling @ 25%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72037 and previous config saved to /var/cache/conftool/dbconfig/20250114-110248-root.json
* 10:47 marostegui@cumin1002: dbctl commit (dc=all): 'es1044 (re)pooling @ 10%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72036 and previous config saved to /var/cache/conftool/dbconfig/20250114-104743-root.json
* 10:45 marostegui@dns1006: END - running authdns-update
* 10:43 marostegui@dns1006: START - running authdns-update
* 10:43 marostegui@dns1006: END - running authdns-update
* 10:41 marostegui@dns1006: START - running authdns-update
* 10:37 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp
* 10:35 marostegui: Reboot db2235 m5 codfw master
* 10:34 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2235.codfw.wmnet with reason: upgrade
* 10:34 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 5:00:00 on db2235.codfw.wmnet with reason: upgrade
* 10:32 marostegui@cumin1002: dbctl commit (dc=all): 'es1044 (re)pooling @ 5%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72035 and previous config saved to /var/cache/conftool/dbconfig/20250114-103238-root.json
* 10:27 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_codfw and A:cp
* 10:17 marostegui@cumin1002: dbctl commit (dc=all): 'es1044 (re)pooling @ 4%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72034 and previous config saved to /var/cache/conftool/dbconfig/20250114-101732-root.json
* 10:05 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_codfw and A:cp
* 10:03 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp
* 10:02 marostegui@cumin1002: dbctl commit (dc=all): 'es1044 (re)pooling @ 3%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72033 and previous config saved to /var/cache/conftool/dbconfig/20250114-100227-root.json
* 10:00 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 09:59 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 09:54 marostegui@cumin1002: dbctl commit (dc=all): 'Repool pc4 T383398', diff saved to https://phabricator.wikimedia.org/P72032 and previous config saved to /var/cache/conftool/dbconfig/20250114-095404-marostegui.json
* 09:53 marostegui@cumin1002: dbctl commit (dc=all): 'Promote pc2014 to codfw pc4 master dbmaint T383398', diff saved to https://phabricator.wikimedia.org/P72031 and previous config saved to /var/cache/conftool/dbconfig/20250114-095320-marostegui.json
* 09:50 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2212-2215].codfw.wmnet
* 09:50 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2212-2215].codfw.wmnet
* 09:47 marostegui@cumin1002: dbctl commit (dc=all): 'es1044 (re)pooling @ 2%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72030 and previous config saved to /var/cache/conftool/dbconfig/20250114-094722-root.json
* 09:47 root@cumin1002: END (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for dbprov2006.codfw.wmnet: Renew puppet certificate - root@cumin1002
* 09:47 jelto: homer 'cr*codfw*' commit 'T377877'
* 09:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on pc[2014-2016].codfw.wmnet,pc1016.eqiad.wmnet with reason: reorganizing pc4
* 09:44 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 5:00:00 on pc[2014-2016].codfw.wmnet,pc1016.eqiad.wmnet with reason: reorganizing pc4
* 09:44 root@cumin1002: START - Cookbook sre.puppet.renew-cert for dbprov2006.codfw.wmnet: Renew puppet certificate - root@cumin1002
* 09:43 jelto: homer 'lsw1-c3-codfw*' commit 'T377877'
* 09:43 marostegui@cumin1002: dbctl commit (dc=all): 'Depool pc4 T383398', diff saved to https://phabricator.wikimedia.org/P72028 and previous config saved to /var/cache/conftool/dbconfig/20250114-094350-marostegui.json
* 09:43 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp
* 09:43 hashar@deploy2002: Finished scap sync-world: Backport for gerrit:1111106knwiki, knwikisource, knwiktionary, knwikiquote: update logo, wordmark (T382802), gerrit:1111109hiwikisource: logo fix (T310961) (duration: 16m 21s)
* 09:38 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2215.codfw.wmnet with OS bookworm
* 09:35 hashar@deploy2002: anzx, hashar: Continuing with sync
* 09:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es[1022,1043].eqiad.wmnet with reason: cloning
* 09:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on es[1022,1043].eqiad.wmnet with reason: cloning
* 09:33 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1022 T382569', diff saved to https://phabricator.wikimedia.org/P72027 and previous config saved to /var/cache/conftool/dbconfig/20250114-093315-marostegui.json
* 09:32 marostegui@cumin1002: dbctl commit (dc=all): 'es1044 (re)pooling @ 1%: Pooling for the first time', diff saved to https://phabricator.wikimedia.org/P72026 and previous config saved to /var/cache/conftool/dbconfig/20250114-093216-root.json
* 09:31 marostegui@cumin1002: dbctl commit (dc=all): 'Add es1044 to dbctl depooled T382569', diff saved to https://phabricator.wikimedia.org/P72025 and previous config saved to /var/cache/conftool/dbconfig/20250114-093147-marostegui.json
* 09:31 hashar@deploy2002: anzx, hashar: Backport for gerrit:1111106knwiki, knwikisource, knwiktionary, knwikiquote: update logo, wordmark (T382802), gerrit:1111109hiwikisource: logo fix (T310961) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 09:26 hashar@deploy2002: Started scap sync-world: Backport for gerrit:1111106knwiki, knwikisource, knwiktionary, knwikiquote: update logo, wordmark (T382802), gerrit:1111109hiwikisource: logo fix (T310961)
* 09:19 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2215.codfw.wmnet with reason: host reimage
* 09:15 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2215.codfw.wmnet with reason: host reimage
* 09:12 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
* 09:11 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
* 09:10 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
* 09:10 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
* 09:09 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply
* 09:09 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply
* 09:09 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 09:08 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 09:06 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 09:05 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 08:59 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2214.codfw.wmnet with OS bookworm
* 08:59 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2215
* 08:59 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2215
* 08:59 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2215.codfw.wmnet with OS bookworm
* 08:58 jelto@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker2215.codfw.wmnet with OS bookworm
* 08:58 marostegui@cumin1002: dbctl commit (dc=all): 'es1023 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72024 and previous config saved to /var/cache/conftool/dbconfig/20250114-085802-root.json
* 08:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2213.codfw.wmnet with OS bookworm
* 08:55 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbprov2006.codfw.wmnet with reason: os upgrade
* 08:55 jynus@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on dbprov2006.codfw.wmnet with reason: os upgrade
* 08:53 gmodena@deploy2002: Finished scap sync-world: Backport for gerrit:1110777Revert^2 "config: remove eventbus instrumentation setting" (duration: 21m 52s)
* 08:51 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2212.codfw.wmnet with OS bookworm
* 08:43 gmodena@deploy2002: otto, gmodena: Continuing with sync
* 08:42 marostegui@cumin1002: dbctl commit (dc=all): 'es1023 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72023 and previous config saved to /var/cache/conftool/dbconfig/20250114-084256-root.json
* 08:40 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2214.codfw.wmnet with reason: host reimage
* 08:38 gmodena@deploy2002: otto, gmodena: Backport for gerrit:1110777Revert^2 "config: remove eventbus instrumentation setting" synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 08:36 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2213.codfw.wmnet with reason: host reimage
* 08:34 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2214.codfw.wmnet with reason: host reimage
* 08:32 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2212.codfw.wmnet with reason: host reimage
* 08:31 gmodena@deploy2002: Started scap sync-world: Backport for gerrit:1110777Revert^2 "config: remove eventbus instrumentation setting"
* 08:31 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2213.codfw.wmnet with reason: host reimage
* 08:27 marostegui@cumin1002: dbctl commit (dc=all): 'es1023 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72022 and previous config saved to /var/cache/conftool/dbconfig/20250114-082751-root.json
* 08:26 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2212.codfw.wmnet with reason: host reimage
* 08:25 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2215
* 08:25 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2215
* 08:25 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2215
* 08:25 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2215.codfw.wmnet 62.32.192.10.in-addr.arpa 2.6.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:25 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2215.codfw.wmnet 62.32.192.10.in-addr.arpa 2.6.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:25 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:25 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2215 - jelto@cumin1002"
* 08:25 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2215 - jelto@cumin1002"
* 08:22 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 08:21 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2215
* 08:21 moritzm: installing perl security updates
* 08:21 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2215.codfw.wmnet with OS bookworm
* 08:19 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2214
* 08:19 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2214
* 08:19 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2214
* 08:19 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2214.codfw.wmnet 61.32.192.10.in-addr.arpa 1.6.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:19 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2214.codfw.wmnet 61.32.192.10.in-addr.arpa 1.6.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:19 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:19 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2214 - jelto@cumin1002"
* 08:19 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2214 - jelto@cumin1002"
* 08:15 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 08:15 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2214
* 08:15 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2214.codfw.wmnet with OS bookworm
* 08:14 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2213
* 08:14 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2213
* 08:14 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2213
* 08:14 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2213.codfw.wmnet 60.32.192.10.in-addr.arpa 0.6.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:14 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2213.codfw.wmnet 60.32.192.10.in-addr.arpa 0.6.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:14 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:14 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2213 - jelto@cumin1002"
* 08:14 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2213 - jelto@cumin1002"
* 08:12 marostegui@cumin1002: dbctl commit (dc=all): 'es1023 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72021 and previous config saved to /var/cache/conftool/dbconfig/20250114-081246-root.json
* 08:10 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 08:10 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2213
* 08:10 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2213.codfw.wmnet with OS bookworm
* 08:09 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2212
* 08:09 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2212
* 08:08 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2212
* 08:08 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2212.codfw.wmnet 59.32.192.10.in-addr.arpa 9.5.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:08 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2212.codfw.wmnet 59.32.192.10.in-addr.arpa 9.5.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 08:08 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:08 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2212 - jelto@cumin1002"
* 08:08 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2212 - jelto@cumin1002"
* 08:05 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 08:02 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2212
* 08:02 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2212.codfw.wmnet with OS bookworm
* 07:57 marostegui@cumin1002: dbctl commit (dc=all): 'es1023 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72020 and previous config saved to /var/cache/conftool/dbconfig/20250114-075741-root.json
* 07:54 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2212.codfw.wmnet wikikube-worker2213.codfw.wmnet wikikube-worker2214.codfw.wmnet wikikube-worker2215.codfw.wmnet on all recursors
* 07:54 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2212.codfw.wmnet wikikube-worker2213.codfw.wmnet wikikube-worker2214.codfw.wmnet wikikube-worker2215.codfw.wmnet on all recursors
* 07:54 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2415 to wikikube-worker2215
* 07:53 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2215
* 07:53 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2215
* 07:53 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2415 to wikikube-worker2215 - jelto@cumin1002"
* 07:52 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2415 to wikikube-worker2215 - jelto@cumin1002"
* 07:49 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 07:49 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2415 to wikikube-worker2215
* 07:48 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2414 to wikikube-worker2214
* 07:48 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2214
* 07:47 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2214
* 07:47 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:47 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2414 to wikikube-worker2214 - jelto@cumin1002"
* 07:47 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2414 to wikikube-worker2214 - jelto@cumin1002"
* 07:43 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 07:43 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2414 to wikikube-worker2214
* 07:40 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2413 to wikikube-worker2213
* 07:39 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2213
* 07:39 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2213
* 07:39 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:39 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2413 to wikikube-worker2213 - jelto@cumin1002"
* 07:39 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2413 to wikikube-worker2213 - jelto@cumin1002"
* 07:36 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 07:35 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2413 to wikikube-worker2213
* 07:34 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2412 to wikikube-worker2212
* 07:34 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2212
* 07:33 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2212
* 07:33 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:33 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2412 to wikikube-worker2212 - jelto@cumin1002"
* 07:33 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2412 to wikikube-worker2212 - jelto@cumin1002"
* 07:29 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 07:29 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2412 to wikikube-worker2212
* 07:24 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[2412-2415].codfw.wmnet
* 07:21 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[2412-2415].codfw.wmnet
* 05:03 mwpresync@deploy2002: Pruned MediaWiki: 1.44.0-wmf.6 (duration: 03m 06s)
* 04:54 mwpresync@deploy2002: Finished scap sync-world: testwikis to 1.44.0-wmf.12 refs T382363 (duration: 50m 57s)
* 04:03 mwpresync@deploy2002: Started scap sync-world: testwikis to 1.44.0-wmf.12 refs T382363
== 2025-01-13 ==
* 22:26 cwhite@deploy2002: Finished deploy [statsv/statsv@42a4331]: T382729 (duration: 00m 08s)
* 22:26 cwhite@deploy2002: Started deploy [statsv/statsv@42a4331]: T382729
* 21:53 bking@cumin2002: conftool action : set/pooled=no; selector: service=(cloudelastic-chi-sslcloudelastic-psi-sslcloudelastic-omega-sslcloudelastic-chi-ssl-publiccloudelastic-psi-ssl-publiccloudelastic-omega-ssl-public),name=cloudelastic1006.eqiad.wmnet
* 21:53 bking@cumin2002: conftool action : set/pooled=no; selector: service=(cloudelastic-chi-sslcloudelastic-psi-sslcloudelastic-omega-sslcloudelastic-chi-ssl-publiccloudelastic-psi-ssl-publiccloudelastic-omega-ssl-public),name=cloudelastic1005.eqiad.wmnet
* 21:50 cdanis@deploy2002: Finished scap sync-world: Backport for gerrit:1110860Add missing parsoid settings for new wikis (T381379 T381080 T378463) (duration: 13m 12s)
* 21:42 cdanis@deploy2002: cdanis, pppery: Continuing with sync
* 21:42 cdanis@deploy2002: cdanis, pppery: Backport for gerrit:1110860Add missing parsoid settings for new wikis (T381379 T381080 T378463) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:37 cdanis@deploy2002: Started scap sync-world: Backport for gerrit:1110860Add missing parsoid settings for new wikis (T381379 T381080 T378463)
* 21:37 cdanis@deploy2002: Finished scap sync-world: Backport for gerrit:1104732Configure new wikis (T381379 T381080 T378463) (duration: 13m 23s)
* 21:29 cdanis@deploy2002: cdanis, pppery: Continuing with sync
* 21:29 cdanis@deploy2002: cdanis, pppery: Backport for gerrit:1104732Configure new wikis (T381379 T381080 T378463) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:24 cdanis@deploy2002: Started scap sync-world: Backport for gerrit:1104732Configure new wikis (T381379 T381080 T378463)
* 21:23 cdanis@deploy2002: Finished scap sync-world: Backport for gerrit:1110814Set Flow to read-only on phase 2a wikis (T378834) (duration: 14m 02s)
* 21:14 cdanis@deploy2002: kemayo, cdanis: Continuing with sync
* 21:14 cdanis@deploy2002: kemayo, cdanis: Backport for gerrit:1110814Set Flow to read-only on phase 2a wikis (T378834) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:09 cdanis@deploy2002: Started scap sync-world: Backport for gerrit:1110814Set Flow to read-only on phase 2a wikis (T378834)
* 19:08 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1097.eqiad.wmnet with OS bookworm
* 19:04 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1096.eqiad.wmnet with OS bookworm
* 18:50 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1097.eqiad.wmnet with reason: host reimage
* 18:47 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1096.eqiad.wmnet with reason: host reimage
* 18:43 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1097.eqiad.wmnet with reason: host reimage
* 18:43 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1096.eqiad.wmnet with reason: host reimage
* 18:28 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1097
* 18:28 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1097
* 18:27 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1097.eqiad.wmnet with OS bookworm
* 18:27 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1096
* 18:27 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1096
* 18:27 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1096.eqiad.wmnet with OS bookworm
* 18:26 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1096.eqiad.wmnet wikikube-worker1097.eqiad.wmnet on all recursors
* 18:26 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1096.eqiad.wmnet wikikube-worker1097.eqiad.wmnet on all recursors
* 18:26 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1446 to wikikube-worker1097
* 18:26 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1097
* 18:25 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1097
* 18:25 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:25 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1446 to wikikube-worker1097 - kamila@cumin1002"
* 18:25 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1446 to wikikube-worker1097 - kamila@cumin1002"
* 18:23 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1445 to wikikube-worker1096
* 18:23 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1096
* 18:22 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1096
* 18:22 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:22 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1445 to wikikube-worker1096 - kamila@cumin1002"
* 18:21 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1445 to wikikube-worker1096 - kamila@cumin1002"
* 18:21 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 18:21 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1446 to wikikube-worker1097
* 18:15 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 18:14 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1445 to wikikube-worker1096
* 17:46 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2208-2211].codfw.wmnet
* 17:46 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2208-2211].codfw.wmnet
* 17:45 jelto: sudo homer 'cr*codfw*' commit 'T377877'
* 17:44 jelto: homer 'lsw1-c3-codfw*' commit 'T377877'
* 17:40 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2211.codfw.wmnet with OS bookworm
* 17:26 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2210.codfw.wmnet with OS bookworm
* 17:25 cdanis@deploy2002: Finished scap sync-world: Backport for gerrit:1109754OpenTelemetry tracing to all wikis (T340552) (duration: 14m 00s)
* 17:19 dzahn@dns1006: END - running authdns-update
* 17:18 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2211.codfw.wmnet with reason: host reimage
* 17:18 mutante: DNS - removing za.wikimedia.org and za.m.wikimedia.org - wiki was not created (T382730, T195926)
* 17:17 dzahn@dns1006: START - running authdns-update
* 17:16 cdanis@deploy2002: cdanis: Continuing with sync
* 17:15 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2209.codfw.wmnet with OS bookworm
* 17:15 cdanis@deploy2002: cdanis: Backport for gerrit:1109754OpenTelemetry tracing to all wikis (T340552) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 17:15 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2211.codfw.wmnet with reason: host reimage
* 17:11 cdanis@deploy2002: Started scap sync-world: Backport for gerrit:1109754OpenTelemetry tracing to all wikis (T340552)
* 17:08 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2208.codfw.wmnet with OS bookworm
* 17:06 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2210.codfw.wmnet with reason: host reimage
* 17:03 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2210.codfw.wmnet with reason: host reimage
* 16:59 root@cumin1002: END (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for dbprov2005.codfw.wmnet: Renew puppet certificate - root@cumin1002
* 16:59 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2211
* 16:58 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2211
* 16:58 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2211
* 16:58 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2211.codfw.wmnet 66.32.192.10.in-addr.arpa 6.6.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 16:58 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2211.codfw.wmnet 66.32.192.10.in-addr.arpa 6.6.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 16:58 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:58 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2211 - jelto@cumin1002"
* 16:58 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2211 - jelto@cumin1002"
* 16:57 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2209.codfw.wmnet with reason: host reimage
* 16:56 root@cumin1002: START - Cookbook sre.puppet.renew-cert for dbprov2005.codfw.wmnet: Renew puppet certificate - root@cumin1002
* 16:54 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 16:54 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2211
* 16:53 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2211.codfw.wmnet with OS bookworm
* 16:52 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2209.codfw.wmnet with reason: host reimage
* 16:48 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2208.codfw.wmnet with reason: host reimage
* 16:46 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2210
* 16:46 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2210
* 16:44 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2208.codfw.wmnet with reason: host reimage
* 16:43 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2210
* 16:43 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2210.codfw.wmnet 65.32.192.10.in-addr.arpa 5.6.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 16:43 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2210.codfw.wmnet 65.32.192.10.in-addr.arpa 5.6.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 16:43 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:43 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2210 - jelto@cumin1002"
* 16:43 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2210 - jelto@cumin1002"
* 16:38 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 16:38 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2210
* 16:38 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2210.codfw.wmnet with OS bookworm
* 16:36 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2209
* 16:36 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2209
* 16:35 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2209
* 16:35 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2209.codfw.wmnet 64.32.192.10.in-addr.arpa 4.6.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 16:35 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2209.codfw.wmnet 64.32.192.10.in-addr.arpa 4.6.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 16:35 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:35 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2209 - jelto@cumin1002"
* 16:35 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2209 - jelto@cumin1002"
* 16:32 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 16:32 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2209
* 16:32 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2209.codfw.wmnet with OS bookworm
* 16:27 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2208
* 16:27 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2208
* 16:27 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2208
* 16:27 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2208.codfw.wmnet 63.32.192.10.in-addr.arpa 3.6.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 16:27 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2208.codfw.wmnet 63.32.192.10.in-addr.arpa 3.6.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 16:27 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:27 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2208 - jelto@cumin1002"
* 16:27 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2208 - jelto@cumin1002"
* 16:23 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 16:23 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2208
* 16:23 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2208.codfw.wmnet with OS bookworm
* 16:20 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2208.codfw.wmnet wikikube-worker2209.codfw.wmnet wikikube-worker2210.codfw.wmnet wikikube-worker2211.codfw.wmnet on all recursors
* 16:20 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2208.codfw.wmnet wikikube-worker2209.codfw.wmnet wikikube-worker2210.codfw.wmnet wikikube-worker2211.codfw.wmnet on all recursors
* 16:20 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2419 to wikikube-worker2211
* 16:19 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2211
* 16:19 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2211
* 16:19 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:19 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2419 to wikikube-worker2211 - jelto@cumin1002"
* 16:18 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2419 to wikikube-worker2211 - jelto@cumin1002"
* 16:14 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 16:14 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2419 to wikikube-worker2211
* 16:14 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2418 to wikikube-worker2210
* 16:13 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2210
* 16:13 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2210
* 16:13 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:13 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2418 to wikikube-worker2210 - jelto@cumin1002"
* 16:12 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2418 to wikikube-worker2210 - jelto@cumin1002"
* 16:09 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 16:09 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2418 to wikikube-worker2210
* 16:06 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2417 to wikikube-worker2209
* 16:05 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2209
* 16:05 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2209
* 16:05 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:05 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2417 to wikikube-worker2209 - jelto@cumin1002"
* 16:04 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2417 to wikikube-worker2209 - jelto@cumin1002"
* 16:01 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 16:01 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2417 to wikikube-worker2209
* 16:00 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2416 to wikikube-worker2208
* 15:59 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2208
* 15:59 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2208
* 15:59 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:59 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2416 to wikikube-worker2208 - jelto@cumin1002"
* 15:58 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2416 to wikikube-worker2208 - jelto@cumin1002"
* 15:55 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 15:55 jelto@cumin1002: START - Cookbook sre.hosts.rename from mw2416 to wikikube-worker2208
* 15:53 mutante: DNS - removing uz.wikimedia.org - wiki was never created (T270987)
* 15:53 dzahn@dns1006: END - running authdns-update
* 15:52 mutante: DNS - removing uz.wikimedia.org - wiki was never created (T382730)
* 15:52 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[2416-2419].codfw.wmnet
* 15:51 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1020 T382569', diff saved to https://phabricator.wikimedia.org/P72015 and previous config saved to /var/cache/conftool/dbconfig/20250113-155153-marostegui.json
* 15:51 dzahn@dns1006: START - running authdns-update
* 15:51 marostegui@cumin1002: dbctl commit (dc=all): 'Set es1041 as es4 eqiad master dbmaint T382569', diff saved to https://phabricator.wikimedia.org/P72014 and previous config saved to /var/cache/conftool/dbconfig/20250113-155135-marostegui.json
* 15:47 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[2416-2419].codfw.wmnet
* 15:44 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbprov2005.codfw.wmnet with reason: os upgrade
* 15:44 jynus@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on dbprov2005.codfw.wmnet with reason: os upgrade
* 15:33 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2206-2207].codfw.wmnet
* 15:32 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2206-2207].codfw.wmnet
* 15:32 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbprov2004.codfw.wmnet with reason: reboot
* 15:32 jynus@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on dbprov2004.codfw.wmnet with reason: reboot
* 15:31 jelto: homer 'cr*codfw*' commit 'T377877'
* 15:30 marostegui@cumin1002: dbctl commit (dc=all): 'Remove db2128 from dbctl T383572', diff saved to https://phabricator.wikimedia.org/P72013 and previous config saved to /var/cache/conftool/dbconfig/20250113-153046-marostegui.json
* 15:30 jelto: homer 'lsw1-c5-codfw*' commit 'T377877'
* 15:28 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2207.codfw.wmnet with OS bookworm
* 15:28 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2128 T383572', diff saved to https://phabricator.wikimedia.org/P72012 and previous config saved to /var/cache/conftool/dbconfig/20250113-152828-marostegui.json
* 15:23 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2234.codfw.wmnet with reason: maintenance
* 15:23 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 5:00:00 on db2234.codfw.wmnet with reason: maintenance
* 15:23 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2206.codfw.wmnet with OS bookworm
* 15:16 cgoubert@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply
* 15:15 cgoubert@deploy2002: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply
* 15:14 cgoubert@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply
* 15:12 cgoubert@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
* 15:09 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2207.codfw.wmnet with reason: host reimage
* 15:06 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2207.codfw.wmnet with reason: host reimage
* 15:03 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2206.codfw.wmnet with reason: host reimage
* 14:59 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2206.codfw.wmnet with reason: host reimage
* 14:59 btullis@deploy2002: Finished deploy [airflow-dags/search@8c96899]: (no justification provided) (duration: 00m 24s)
* 14:58 btullis@deploy2002: Started deploy [airflow-dags/search@8c96899]: (no justification provided)
* 14:54 marostegui@cumin1002: dbctl commit (dc=all): 'db2240 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72011 and previous config saved to /var/cache/conftool/dbconfig/20250113-145432-root.json
* 14:52 Lucas_WMDE: UTC afternoon backport+config window done
* 14:51 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1109832prod: Enable $wgCampaignEventsEnableEventWikis (T380078) (duration: 32m 04s)
* 14:49 moritzm: installing glibc bugfix updates for Bookworm
* 14:48 jayme: re-enabling puppet on all hosts running kubelet - T383413
* 14:47 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2207
* 14:47 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2207
* 14:46 otto@deploy2002: Finished deploy [analytics/refinery@f3945ee]: gobblin eventlogging_legacy - use EventStreamConfig to pull topics (duration: 01m 27s)
* 14:45 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2207
* 14:45 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2207.codfw.wmnet 166.32.192.10.in-addr.arpa 6.6.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:45 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2207.codfw.wmnet 166.32.192.10.in-addr.arpa 6.6.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:45 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:45 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2207 - jelto@cumin1002"
* 14:45 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2207 - jelto@cumin1002"
* 14:44 otto@deploy2002: Started deploy [analytics/refinery@f3945ee]: gobblin eventlogging_legacy - use EventStreamConfig to pull topics
* 14:43 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, daimona: Continuing with sync
* 14:42 dcausse@deploy2002: Finished deploy [airflow-dags/search@8c96899]: search: fix glent, import_cirrus_indexes and transfer_to_es (duration: 01m 44s)
* 14:42 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 14:42 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2207
* 14:41 jayme: disabling puppet on all hosts running kubelet - T383413
* 14:41 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2206
* 14:41 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2206
* 14:41 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2206
* 14:41 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2206.codfw.wmnet 167.32.192.10.in-addr.arpa 7.6.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:41 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2206.codfw.wmnet 167.32.192.10.in-addr.arpa 7.6.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:41 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:41 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2206 - jelto@cumin1002"
* 14:41 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2206 - jelto@cumin1002"
* 14:41 otto@deploy2002: Finished deploy [analytics/refinery@f3945ee] (hadoop-test): gobblin eventlogging_legacy - use EventStreamConfig to pull topics (duration: 01m 37s)
* 14:40 dcausse@deploy2002: Started deploy [airflow-dags/search@8c96899]: search: fix glent, import_cirrus_indexes and transfer_to_es
* 14:40 otto@deploy2002: Started deploy [analytics/refinery@f3945ee] (hadoop-test): gobblin eventlogging_legacy - use EventStreamConfig to pull topics
* 14:39 marostegui@cumin1002: dbctl commit (dc=all): 'db2240 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72010 and previous config saved to /var/cache/conftool/dbconfig/20250113-143926-root.json
* 14:37 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 14:37 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2206
* 14:37 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2207.codfw.wmnet with OS bookworm
* 14:37 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2206.codfw.wmnet with OS bookworm
* 14:31 btullis@dns1004: END - running authdns-update
* 14:30 btullis@dns1004: START - running authdns-update
* 14:26 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2206.codfw.wmnet wikikube-worker2207.codfw.wmnet on all recursors
* 14:26 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2206.codfw.wmnet wikikube-worker2207.codfw.wmnet on all recursors
* 14:25 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts eventlog1003.eqiad.wmnet
* 14:25 btullis@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:24 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, daimona: Backport for gerrit:1109832prod: Enable $wgCampaignEventsEnableEventWikis (T380078) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2240 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72009 and previous config saved to /var/cache/conftool/dbconfig/20250113-142421-root.json
* 14:24 root@cumin1002: END (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for dbprov2003.codfw.wmnet: Renew puppet certificate - root@cumin1002
* 14:24 klausman@deploy2002: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
* 14:23 klausman@deploy2002: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
* 14:22 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2041 to wikikube-worker2207
* 14:22 btullis@cumin1002: START - Cookbook sre.dns.netbox
* 14:22 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2207
* 14:21 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2207
* 14:21 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:21 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2041 to wikikube-worker2207 - jelto@cumin1002"
* 14:21 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2041 to wikikube-worker2207 - jelto@cumin1002"
* 14:21 root@cumin1002: START - Cookbook sre.puppet.renew-cert for dbprov2003.codfw.wmnet: Renew puppet certificate - root@cumin1002
* 14:19 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1109832prod: Enable $wgCampaignEventsEnableEventWikis (T380078)
* 14:18 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2040 to wikikube-worker2206
* 14:18 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2206
* 14:18 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 14:18 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2206
* 14:18 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:17 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2040 to wikikube-worker2206 - jelto@cumin1002"
* 14:17 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2040 to wikikube-worker2206 - jelto@cumin1002"
* 14:16 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2041 to wikikube-worker2207
* 14:14 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 14:13 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2040 to wikikube-worker2206
* 14:09 marostegui@cumin1002: dbctl commit (dc=all): 'db2240 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P72008 and previous config saved to /var/cache/conftool/dbconfig/20250113-140916-root.json
* 14:08 klausman@deploy2002: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
* 14:07 klausman@deploy2002: helmfile [codfw] START helmfile.d/services/api-gateway: apply
* 14:07 btullis@cumin1002: START - Cookbook sre.hosts.decommission for hosts eventlog1003.eqiad.wmnet
* 14:06 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes[2040-2041].codfw.wmnet
* 14:05 klausman@deploy2002: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
* 14:04 klausman@deploy2002: helmfile [staging] START helmfile.d/services/api-gateway: apply
* 14:04 filippo@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-query,name=eqiad
* 14:02 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes[2040-2041].codfw.wmnet
* 13:54 marostegui@cumin1002: dbctl commit (dc=all): 'db2240 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72006 and previous config saved to /var/cache/conftool/dbconfig/20250113-135410-root.json
* 13:52 kamila_: homer cr*eqiad* commit 'wikikube decoms'
* 13:48 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wikikube-worker[1013-1014,1017-1018].eqiad.wmnet
* 13:48 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:48 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wikikube-worker[1013-1014,1017-1018].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - kamila@cumin1002"
* 13:48 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wikikube-worker[1013-1014,1017-1018].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - kamila@cumin1002"
* 13:41 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 13:38 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db[2134,2160,2234].codfw.wmnet with reason: maintenance
* 13:38 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 10:00:00 on db[2134,2160,2234].codfw.wmnet with reason: maintenance
* 13:28 filippo@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=thanos-query,name=eqiad
* 13:27 filippo@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-swift,name=eqiad
* 13:27 filippo@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=thanos-swift,name=eqiad
* 13:25 godog: bounce thanos-store on titan1*
* 13:24 godog: bounce thanos-query on titan1*
* 13:19 kamila@cumin1002: START - Cookbook sre.hosts.decommission for hosts wikikube-worker[1013-1014,1017-1018].eqiad.wmnet
* 13:18 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wikikube-worker[1008-1010].eqiad.wmnet
* 13:18 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:18 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wikikube-worker[1008-1010].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - kamila@cumin1002"
* 13:18 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wikikube-worker[1008-1010].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - kamila@cumin1002"
* 13:13 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 13:10 Daimona: Creating new DB tables for the CampaignEvents extension in x1.testwiki, x1.test2wiki, x1.officewiki, and x1.wikishared # T379294 T381424
* 12:53 kamila@cumin1002: START - Cookbook sre.hosts.decommission for hosts wikikube-worker[1008-1010].eqiad.wmnet
* 12:50 kamila@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts wikikube-worker[1008-1010].eqiad.wmnet
* 12:49 kamila@cumin1002: START - Cookbook sre.hosts.decommission for hosts wikikube-worker[1008-1010].eqiad.wmnet
* 12:27 marostegui@dns1006: END - running authdns-update
* 12:26 marostegui@dns1006: START - running authdns-update
* 12:25 marostegui: Switch m2-master proxy
* 12:21 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1008-1010,1013-1014,1017-1018].eqiad.wmnet
* 12:18 reedy@deploy2002: Synchronized php-1.44.0-wmf.11/extensions/UploadWizard/: T383415 (duration: 13m 05s)
* 12:17 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1008-1010,1013-1014,1017-1018].eqiad.wmnet
* 11:58 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2203-2205].codfw.wmnet
* 11:58 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2203-2205].codfw.wmnet
* 11:57 jayme: re-enabling puppet on all hosts running kubelet - T383413
* 11:53 jelto: homer 'cr*codfw*' commit 'T377877'
* 11:53 jelto: homer 'lsw1-d1-codfw*' commit 'T377877'
* 11:51 jelto: homer 'lsw1-c6-codfw*' commit 'T377877'
* 11:51 jayme: disabling puppet on all hosts running kubelet - T383413
* 11:50 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2205.codfw.wmnet with OS bookworm
* 11:50 stran@deploy2002: helmfile [codfw] DONE helmfile.d/services/ipoid: apply
* 11:49 stran@deploy2002: helmfile [codfw] START helmfile.d/services/ipoid: apply
* 11:49 stran@deploy2002: helmfile [eqiad] DONE helmfile.d/services/ipoid: apply
* 11:48 marostegui@cumin1002: dbctl commit (dc=all): 'db1170 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P72004 and previous config saved to /var/cache/conftool/dbconfig/20250113-114852-root.json
* 11:48 stran@deploy2002: helmfile [eqiad] START helmfile.d/services/ipoid: apply
* 11:48 stran@deploy2002: helmfile [staging] DONE helmfile.d/services/ipoid: apply
* 11:46 stran@deploy2002: helmfile [staging] START helmfile.d/services/ipoid: apply
* 11:45 jayme@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:45 jayme@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jayme@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:45 jayme@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:45 jayme@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 11:45 jayme@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 11:45 jayme@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 11:45 jayme@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 11:45 jayme@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 11:44 jayme@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 11:44 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 11:44 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 11:44 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 11:44 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 11:44 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 11:44 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 11:44 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 11:44 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 11:43 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2204.codfw.wmnet with OS bookworm
* 11:33 marostegui@cumin1002: dbctl commit (dc=all): 'db1170 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P72002 and previous config saved to /var/cache/conftool/dbconfig/20250113-113347-root.json
* 11:31 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2205.codfw.wmnet with reason: host reimage
* 11:28 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2205.codfw.wmnet with reason: host reimage
* 11:26 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2203.codfw.wmnet with OS bookworm
* 11:24 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db[2132,2160,2232].codfw.wmnet with reason: maintenance
* 11:24 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 10:00:00 on db[2132,2160,2232].codfw.wmnet with reason: maintenance
* 11:24 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2204.codfw.wmnet with reason: host reimage
* 11:20 marostegui: Move db2160:3322 under db2232 in m2 codfw dbmaint T373579
* 11:20 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2204.codfw.wmnet with reason: host reimage
* 11:19 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2132,2160,2232].codfw.wmnet with reason: maintenance
* 11:18 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on db[2132,2160,2232].codfw.wmnet with reason: maintenance
* 11:18 marostegui@cumin1002: dbctl commit (dc=all): 'db1170 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P72000 and previous config saved to /var/cache/conftool/dbconfig/20250113-111842-root.json
* 11:09 moritzm: installing pymysql security updates
* 11:09 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2205
* 11:09 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2205
* 11:07 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2203.codfw.wmnet with reason: host reimage
* 11:07 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2205
* 11:07 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2205.codfw.wmnet 230.48.192.10.in-addr.arpa 0.3.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 11:06 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2205.codfw.wmnet 230.48.192.10.in-addr.arpa 0.3.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 11:06 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:06 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2205 - jelto@cumin1002"
* 11:06 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2205 - jelto@cumin1002"
* 11:05 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db[2133,2160].codfw.wmnet with reason: cloning
* 11:05 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 10:00:00 on db[2133,2160].codfw.wmnet with reason: cloning
* 11:04 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2203.codfw.wmnet with reason: host reimage
* 11:03 marostegui@cumin1002: dbctl commit (dc=all): 'db1170 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P71998 and previous config saved to /var/cache/conftool/dbconfig/20250113-110336-root.json
* 11:03 marostegui@cumin1002: dbctl commit (dc=all): 'Remove db2123 from dbctl for decommission', diff saved to https://phabricator.wikimedia.org/P71997 and previous config saved to /var/cache/conftool/dbconfig/20250113-110333-marostegui.json
* 11:03 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 11:03 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2205
* 11:03 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2205.codfw.wmnet with OS bookworm
* 11:01 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2204
* 11:01 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2204
* 11:01 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2204
* 11:01 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2204.codfw.wmnet 164.32.192.10.in-addr.arpa 4.6.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 11:01 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2204.codfw.wmnet 164.32.192.10.in-addr.arpa 4.6.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 11:01 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:01 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2204 - jelto@cumin1002"
* 11:01 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2204 - jelto@cumin1002"
* 11:00 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2003.codfw.wmnet with reason: os upgrade
* 11:00 jynus@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2003.codfw.wmnet with reason: os upgrade
* 10:58 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 10:57 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2204
* 10:57 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2204.codfw.wmnet with OS bookworm
* 10:55 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2123.codfw.wmnet
* 10:55 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:55 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2123.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 10:54 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2123.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 10:50 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 10:49 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2240.codfw.wmnet
* 10:46 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2123.codfw.wmnet
* 10:45 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2203
* 10:45 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2203
* 10:45 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2203
* 10:45 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2203.codfw.wmnet 165.32.192.10.in-addr.arpa 5.6.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:44 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2203.codfw.wmnet 165.32.192.10.in-addr.arpa 5.6.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:44 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:44 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2203 - jelto@cumin1002"
* 10:44 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2203 - jelto@cumin1002"
* 10:43 root@cumin1002: START - Cookbook sre.mysql.upgrade for db2240.codfw.wmnet
* 10:43 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2240', diff saved to https://phabricator.wikimedia.org/P71996 and previous config saved to /var/cache/conftool/dbconfig/20250113-104310-marostegui.json
* 10:42 marostegui@cumin1002: dbctl commit (dc=all): 'db2240 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P71995 and previous config saved to /var/cache/conftool/dbconfig/20250113-104250-root.json
* 10:41 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 10:41 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2203
* 10:41 marostegui@cumin1002: dbctl commit (dc=all): 'db1170 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P71994 and previous config saved to /var/cache/conftool/dbconfig/20250113-104115-root.json
* 10:41 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2203.codfw.wmnet with OS bookworm
* 10:36 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2203.codfw.wmnet wikikube-worker2204.codfw.wmnet wikikube-worker2205.codfw.wmnet on all recursors
* 10:36 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2203.codfw.wmnet wikikube-worker2204.codfw.wmnet wikikube-worker2205.codfw.wmnet on all recursors
* 10:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on pc1015.eqiad.wmnet with reason: cloning
* 10:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 5:00:00 on pc1015.eqiad.wmnet with reason: cloning
* 10:28 stran@deploy2002: helmfile [eqiad] DONE helmfile.d/services/ipoid: apply
* 10:28 stran@deploy2002: helmfile [eqiad] START helmfile.d/services/ipoid: apply
* 10:26 stran@deploy2002: helmfile [staging] DONE helmfile.d/services/ipoid: apply
* 10:26 stran@deploy2002: helmfile [staging] START helmfile.d/services/ipoid: apply
* 10:25 stran@deploy2002: helmfile [codfw] DONE helmfile.d/services/ipoid: apply
* 10:25 stran@deploy2002: helmfile [codfw] START helmfile.d/services/ipoid: apply
* 10:25 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2044 to wikikube-worker2205
* 10:24 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2205
* 10:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repool pc3 T383398', diff saved to https://phabricator.wikimedia.org/P71993 and previous config saved to /var/cache/conftool/dbconfig/20250113-102343-marostegui.json
* 10:21 marostegui@cumin1002: dbctl commit (dc=all): 'Make pc1013 master in pc3 T383398', diff saved to https://phabricator.wikimedia.org/P71992 and previous config saved to /var/cache/conftool/dbconfig/20250113-102152-marostegui.json
* 10:20 marostegui@cumin1002: dbctl commit (dc=all): 'Remove pc1015 from pc3', diff saved to https://phabricator.wikimedia.org/P71991 and previous config saved to /var/cache/conftool/dbconfig/20250113-102047-marostegui.json
* 10:19 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2205
* 10:18 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:18 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2044 to wikikube-worker2205 - jelto@cumin1002"
* 10:18 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2044 to wikikube-worker2205 - jelto@cumin1002"
* 10:14 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 10:14 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2044 to wikikube-worker2205
* 10:14 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2043 to wikikube-worker2204
* 10:13 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2204
* 10:13 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for pc1015.eqiad.wmnet
* 10:13 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2204
* 10:13 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:13 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2043 to wikikube-worker2204 - jelto@cumin1002"
* 10:12 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for pc2013.codfw.wmnet
* 10:12 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2043 to wikikube-worker2204 - jelto@cumin1002"
* 10:11 marostegui@cumin1002: dbctl commit (dc=all): 'Fix weights in pc3', diff saved to https://phabricator.wikimedia.org/P71990 and previous config saved to /var/cache/conftool/dbconfig/20250113-101132-marostegui.json
* 10:10 ladsgroup@deploy2002: Finished scap sync-world: Backport for gerrit:1109752Add wikitech.wikimedia.org to list of local vhosts (T376305) (duration: 22m 28s)
* 10:08 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 10:07 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2043 to wikikube-worker2204
* 10:07 marostegui: Upgrade pc2013 pc1015 pc3 dbmaint eqiad codfw T383398
* 10:07 root@cumin1002: START - Cookbook sre.mysql.upgrade for pc1015.eqiad.wmnet
* 10:06 root@cumin1002: START - Cookbook sre.mysql.upgrade for pc2013.codfw.wmnet
* 10:05 marostegui@cumin1002: dbctl commit (dc=all): 'Depool pc3 T383398', diff saved to https://phabricator.wikimedia.org/P71989 and previous config saved to /var/cache/conftool/dbconfig/20250113-100554-marostegui.json
* 10:05 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2042 to wikikube-worker2203
* 10:04 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2203
* 10:04 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2203
* 10:04 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:04 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2042 to wikikube-worker2203 - jelto@cumin1002"
* 10:03 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2042 to wikikube-worker2203 - jelto@cumin1002"
* 10:03 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on pc2013.codfw.wmnet with reason: cloning
* 10:02 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 5:00:00 on pc2013.codfw.wmnet with reason: cloning
* 10:02 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on pc1013.eqiad.wmnet with reason: cloning
* 10:02 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 5:00:00 on pc1013.eqiad.wmnet with reason: cloning
* 10:01 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on pc1015.eqiad.wmnet with reason: cloning
* 10:01 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 5:00:00 on pc1015.eqiad.wmnet with reason: cloning
* 10:00 ladsgroup@deploy2002: ladsgroup: Continuing with sync
* 10:00 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 10:00 ladsgroup@deploy2002: ladsgroup: Backport for gerrit:1109752Add wikitech.wikimedia.org to list of local vhosts (T376305) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 09:59 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2042 to wikikube-worker2203
* 09:56 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_ulsfo and not P{cp4052.ulsfo.wmnet} and A:cp
* 09:55 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes[2042-2044].codfw.wmnet
* 09:51 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes[2042-2044].codfw.wmnet
* 09:49 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2212.codfw.wmnet with reason: Reboot
* 09:49 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es1023.eqiad.wmnet with reason: cloning
* 09:49 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2212.codfw.wmnet with reason: Reboot
* 09:49 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on es1023.eqiad.wmnet with reason: cloning
* 09:48 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1023', diff saved to https://phabricator.wikimedia.org/P71988 and previous config saved to /var/cache/conftool/dbconfig/20250113-094846-marostegui.json
* 09:48 marostegui@cumin1002: dbctl commit (dc=all): 'Switchover es5 eqiad master dbmaint T382569', diff saved to https://phabricator.wikimedia.org/P71987 and previous config saved to /var/cache/conftool/dbconfig/20250113-094833-marostegui.json
* 09:48 ladsgroup@deploy2002: Started scap sync-world: Backport for gerrit:1109752Add wikitech.wikimedia.org to list of local vhosts (T376305)
* 09:47 ladsgroup@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2212.codfw.wmnet
* 09:42 ladsgroup@cumin1002: START - Cookbook sre.mysql.upgrade for db2212.codfw.wmnet
* 09:41 Amir1: dbmaint on pc5@eqiad (T382948)
* 09:38 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_ulsfo and not P{cp4052.ulsfo.wmnet} and A:cp
* 09:36 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_ulsfo and not P{cp4044.ulsfo.wmnet} and A:cp
* 09:17 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_ulsfo and not P{cp4044.ulsfo.wmnet} and A:cp
* 08:57 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2135.codfw.wmnet
* 08:57 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:57 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2135.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 08:57 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2135.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 08:53 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 08:49 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2135.codfw.wmnet
* 08:34 hashar@deploy2002: Finished deploy [integration/docroot@a81d82c]: build: Updating mediawiki/mediawiki-phan-config to 0.15.1 (duration: 00m 09s)
* 08:34 hashar@deploy2002: Started deploy [integration/docroot@a81d82c]: build: Updating mediawiki/mediawiki-phan-config to 0.15.1
* 08:27 moritzm: updated netboot image for bookworm to 12.9 T383537
== 2025-01-12 ==
* 09:13 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 09:12 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
== 2025-01-11 ==
* 23:02 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2212.codfw.wmnet with reason: Replication lag
* 23:02 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on db2212.codfw.wmnet with reason: Replication lag
* 23:02 fabfur@cumin1002: dbctl commit (dc=all): 'Depool db2212', diff saved to https://phabricator.wikimedia.org/P71985 and previous config saved to /var/cache/conftool/dbconfig/20250111-230213-fabfur.json
* 01:39 eileen: config revision changed from to 
* 00:06 eileen: config revision changed from to 
== 2025-01-10 ==
* 23:33 eileen: config revision changed from to 
* 23:09 eileen: config revision changed from to 
* 22:59 eileen: config revision changed from to 
* 22:49 eileen: config revision changed from to 
* 22:45 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: cloudelastic1005*,cloudelastic1006* for ban hosts prior to decom - bking@cumin2002 - T380937
* 22:45 bking@cumin2002: START - Cookbook sre.elasticsearch.ban Banning hosts: cloudelastic1005*,cloudelastic1006* for ban hosts prior to decom - bking@cumin2002 - T380937
* 22:45 bking@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.ban (exit_code=99) Banning hosts: cloudelastic1005,cloudelastic1006 for ban hosts prior to decom - bking@cumin2002 - T380937
* 22:45 bking@cumin2002: START - Cookbook sre.elasticsearch.ban Banning hosts: cloudelastic1005,cloudelastic1006 for ban hosts prior to decom - bking@cumin2002 - T380937
* 22:21 eileen: config revision changed from to 
* 22:11 eileen: config revision changed from to 
* 20:01 eevans@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-codfw: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002
* 19:43 eevans@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-codfw: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002
* 19:41 eevans@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-eqiad: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002
* 19:26 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on eventlog1003.eqiad.wmnet with reason: Shutting down VM in preparation for decommissioning
* 19:26 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on eventlog1003.eqiad.wmnet with reason: Shutting down VM in preparation for decommissioning
* 19:23 eevans@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-eqiad: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002
* 18:49 sukhe: sudo cumin 'P:Mediawiki::Maintenance' 'run-puppet-agent': CR 
* 16:56 cmooney@dns2005: END - running authdns-update
* 16:55 cmooney@dns2005: START - running authdns-update
* 16:50 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:50 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns names in newly assigned wmcs private ipv6 ranges - cmooney@cumin1002"
* 16:50 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns names in newly assigned wmcs private ipv6 ranges - cmooney@cumin1002"
* 16:47 cmooney@cumin1002: START - Cookbook sre.dns.netbox
* 16:28 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:28 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns names in newly assigned wmcs private ipv6 ranges - cmooney@cumin1002"
* 16:28 eevans@deploy2002: helmfile [codfw] DONE helmfile.d/services/data-gateway: apply
* 16:28 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns names in newly assigned wmcs private ipv6 ranges - cmooney@cumin1002"
* 16:27 eevans@deploy2002: helmfile [codfw] START helmfile.d/services/data-gateway: apply
* 16:26 eevans@deploy2002: helmfile [eqiad] DONE helmfile.d/services/data-gateway: apply
* 16:25 eevans@deploy2002: helmfile [eqiad] START helmfile.d/services/data-gateway: apply
* 16:23 cmooney@cumin1002: START - Cookbook sre.dns.netbox
* 16:23 cmooney@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 16:19 cmooney@cumin1002: START - Cookbook sre.dns.netbox
* 15:46 eevans@deploy2002: helmfile [staging] DONE helmfile.d/services/data-gateway: apply
* 15:45 eevans@deploy2002: helmfile [staging] START helmfile.d/services/data-gateway: apply
* 15:36 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2199-2202].codfw.wmnet
* 15:36 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2199-2202].codfw.wmnet
* 15:34 jelto: homer 'cr*codfw*' commit 'T377877'
* 15:33 jelto: homer 'lsw1-d1-codfw*' commit 'T377877'
* 15:33 jelto: homer 'lsw1-d5-codfw*' commit 'T377877'
* 15:32 jelto: homer 'lsw1-d3-codfw*' commit 'T377877'
* 15:32 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2202.codfw.wmnet with OS bookworm
* 15:25 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2201.codfw.wmnet with OS bookworm
* 15:25 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2240.codfw.wmnet with reason: maintenance
* 15:25 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 10:00:00 on db2240.codfw.wmnet with reason: maintenance
* 15:21 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2240.codfw.wmnet with reason: maintenance
* 15:20 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on db2240.codfw.wmnet with reason: maintenance
* 15:20 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2240 to make it candidate master', diff saved to https://phabricator.wikimedia.org/P71984 and previous config saved to /var/cache/conftool/dbconfig/20250110-152035-marostegui.json
* 15:12 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2202.codfw.wmnet with reason: host reimage
* 15:08 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2202.codfw.wmnet with reason: host reimage
* 15:06 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2201.codfw.wmnet with reason: host reimage
* 15:02 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2201.codfw.wmnet with reason: host reimage
* 14:49 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2202
* 14:49 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2202
* 14:48 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2202
* 14:48 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2202.codfw.wmnet 226.48.192.10.in-addr.arpa 6.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:48 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2202.codfw.wmnet 226.48.192.10.in-addr.arpa 6.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:48 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:46 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 14:46 jelto@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 14:46 jelto@cumin1002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2202 - jelto@cumin1002"
* 14:42 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2201
* 14:42 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2201
* 14:42 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2201
* 14:42 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2201.codfw.wmnet 227.48.192.10.in-addr.arpa 7.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:42 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2201.codfw.wmnet 227.48.192.10.in-addr.arpa 7.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 14:42 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:42 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2201 - jelto@cumin1002"
* 14:33 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2202 - jelto@cumin1002"
* 14:29 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 14:27 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2201 - jelto@cumin1002"
* 14:26 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2202
* 14:26 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2202.codfw.wmnet with OS bookworm
* 14:26 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2200.codfw.wmnet with OS bookworm
* 14:24 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 14:24 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2201
* 14:23 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2201.codfw.wmnet with OS bookworm
* 14:23 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2199.codfw.wmnet with OS bookworm
* 14:06 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2200.codfw.wmnet with reason: host reimage
* 14:03 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2199.codfw.wmnet with reason: host reimage
* 14:02 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2200.codfw.wmnet with reason: host reimage
* 13:59 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2199.codfw.wmnet with reason: host reimage
* 13:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: maintenance
* 13:58 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: maintenance
* 13:56 aqu@deploy2002: Finished deploy [airflow-dags/analytics@7a1a552]: Backfill 2024 12: cassandra_load_pageview_per_article (duration: 01m 19s)
* 13:54 aqu@deploy2002: Started deploy [airflow-dags/analytics@7a1a552]: Backfill 2024 12: cassandra_load_pageview_per_article
* 13:54 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1170.eqiad.wmnet with reason: maintenance
* 13:54 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1170.eqiad.wmnet with reason: maintenance
* 13:49 jynus@cumin1002: dbctl commit (dc=all): 'depool db1170', diff saved to https://phabricator.wikimedia.org/P71983 and previous config saved to /var/cache/conftool/dbconfig/20250110-134954-jynus.json
* 13:43 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2200
* 13:43 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2200
* 13:43 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2200
* 13:43 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2200.codfw.wmnet 228.48.192.10.in-addr.arpa 8.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:43 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2200.codfw.wmnet 228.48.192.10.in-addr.arpa 8.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:43 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:40 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:40 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2199
* 13:40 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2199
* 13:40 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2199
* 13:40 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2199.codfw.wmnet 229.48.192.10.in-addr.arpa 9.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:40 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2199.codfw.wmnet 229.48.192.10.in-addr.arpa 9.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:40 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:40 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2199 - jelto@cumin1002"
* 13:40 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2199 - jelto@cumin1002"
* 13:38 marostegui: Move pc2013 to pc4 dbmaint codfw - T383398
* 13:36 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:35 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2200
* 13:35 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2199
* 13:35 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2200.codfw.wmnet with OS bookworm
* 13:34 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2199.codfw.wmnet with OS bookworm
* 13:32 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2199.codfw.wmnet wikikube-worker2200.codfw.wmnet wikikube-worker2201.codfw.wmnet wikikube-worker2202.codfw.wmnet on all recursors
* 13:32 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2199.codfw.wmnet wikikube-worker2200.codfw.wmnet wikikube-worker2201.codfw.wmnet wikikube-worker2202.codfw.wmnet on all recursors
* 13:30 marostegui: Move pc1013 to pc3 dbmaint eqiad - T383398
* 13:30 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2048 to wikikube-worker2202
* 13:29 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2202
* 13:29 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2202
* 13:29 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:29 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2048 to wikikube-worker2202 - jelto@cumin1002"
* 13:28 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2048 to wikikube-worker2202 - jelto@cumin1002"
* 13:25 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:25 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2048 to wikikube-worker2202
* 13:23 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2047 to wikikube-worker2201
* 13:23 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2201
* 13:22 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2201
* 13:22 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:22 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2047 to wikikube-worker2201 - jelto@cumin1002"
* 13:21 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2047 to wikikube-worker2201 - jelto@cumin1002"
* 13:18 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:17 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2047 to wikikube-worker2201
* 13:17 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2046 to wikikube-worker2200
* 13:16 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2200
* 13:15 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2200
* 13:15 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:14 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2046 to wikikube-worker2200 - jelto@cumin1002"
* 13:10 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:10 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2046 to wikikube-worker2200
* 13:09 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2045 to wikikube-worker2199
* 13:09 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2199
* 13:08 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2199
* 13:08 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:08 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2045 to wikikube-worker2199 - jelto@cumin1002"
* 13:08 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2045 to wikikube-worker2199 - jelto@cumin1002"
* 13:05 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 13:04 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2045 to wikikube-worker2199
* 13:00 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes[2045-2048].codfw.wmnet
* 12:58 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes[2045-2048].codfw.wmnet
* 12:44 marostegui@cumin1002: dbctl commit (dc=all): 'db2192 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P71982 and previous config saved to /var/cache/conftool/dbconfig/20250110-124438-root.json
* 12:30 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2126.codfw.wmnet
* 12:30 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:30 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2126.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 12:30 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2126.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 12:29 marostegui@cumin1002: dbctl commit (dc=all): 'db2192 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P71981 and previous config saved to /var/cache/conftool/dbconfig/20250110-122933-root.json
* 12:27 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 12:22 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2126.codfw.wmnet
* 12:22 marostegui@cumin1002: dbctl commit (dc=all): 'Remove db2126 from dbctl T383395', diff saved to https://phabricator.wikimedia.org/P71980 and previous config saved to /var/cache/conftool/dbconfig/20250110-122206-marostegui.json
* 12:16 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2126 T383395', diff saved to https://phabricator.wikimedia.org/P71979 and previous config saved to /var/cache/conftool/dbconfig/20250110-121657-marostegui.json
* 12:14 marostegui@cumin1002: dbctl commit (dc=all): 'db2192 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P71978 and previous config saved to /var/cache/conftool/dbconfig/20250110-121427-root.json
* 12:07 jelto@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab to new version
* 11:59 marostegui@cumin1002: dbctl commit (dc=all): 'db2192 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P71977 and previous config saved to /var/cache/conftool/dbconfig/20250110-115922-root.json
* 11:54 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2195-2198].codfw.wmnet
* 11:54 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2195-2198].codfw.wmnet
* 11:51 jelto: homer 'cr*codw*' commit 'T377877'
* 11:51 jelto: homer 'cr*eqiad*' commit 'T377876'
* 11:50 jelto: homer 'lsw1-d6-codfw*' commit 'T377877'
* 11:50 jelto: homer 'lsw1-d8-codfw*' commit 'T377877'
* 11:49 jelto: homer 'lsw1-d5-codfw*' commit 'T377877'
* 11:46 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2197.codfw.wmnet with OS bookworm
* 11:44 marostegui@cumin1002: dbctl commit (dc=all): 'db2192 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P71976 and previous config saved to /var/cache/conftool/dbconfig/20250110-114417-root.json
* 11:41 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2198.codfw.wmnet with OS bookworm
* 11:26 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2197.codfw.wmnet with reason: host reimage
* 11:22 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2198.codfw.wmnet with reason: host reimage
* 11:21 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2197.codfw.wmnet with reason: host reimage
* 11:19 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2198.codfw.wmnet with reason: host reimage
* 11:06 marostegui@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P71975 and previous config saved to /var/cache/conftool/dbconfig/20250110-110643-root.json
* 11:06 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P71974 and previous config saved to /var/cache/conftool/dbconfig/20250110-110633-root.json
* 11:02 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2197
* 11:02 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2197
* 11:01 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2197
* 11:01 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2197.codfw.wmnet 223.48.192.10.in-addr.arpa 3.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 11:01 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2197.codfw.wmnet 223.48.192.10.in-addr.arpa 3.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 11:01 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:59 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2198
* 10:59 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2198
* 10:59 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 10:58 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2198
* 10:58 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2198.codfw.wmnet 222.48.192.10.in-addr.arpa 2.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:58 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2198.codfw.wmnet 222.48.192.10.in-addr.arpa 2.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:58 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:58 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2198 - jelto@cumin1002"
* 10:58 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2198 - jelto@cumin1002"
* 10:55 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2192.codfw.wmnet with reason: maintenance
* 10:55 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on db2192.codfw.wmnet with reason: maintenance
* 10:55 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2192 to change binlog format', diff saved to https://phabricator.wikimedia.org/P71973 and previous config saved to /var/cache/conftool/dbconfig/20250110-105514-marostegui.json
* 10:55 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 10:54 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2198
* 10:54 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2197
* 10:54 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2198.codfw.wmnet with OS bookworm
* 10:54 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2197.codfw.wmnet with OS bookworm
* 10:52 marostegui@cumin1002: dbctl commit (dc=all): 'es1022 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P71972 and previous config saved to /var/cache/conftool/dbconfig/20250110-105202-root.json
* 10:51 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2195.codfw.wmnet with OS bookworm
* 10:51 marostegui@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P71971 and previous config saved to /var/cache/conftool/dbconfig/20250110-105137-root.json
* 10:51 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P71970 and previous config saved to /var/cache/conftool/dbconfig/20250110-105127-root.json
* 10:49 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2196.codfw.wmnet with OS bookworm
* 10:47 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2123 T383388', diff saved to https://phabricator.wikimedia.org/P71969 and previous config saved to /var/cache/conftool/dbconfig/20250110-104739-marostegui.json
* 10:36 marostegui@cumin1002: dbctl commit (dc=all): 'es1022 (re)pooling @ 75%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P71968 and previous config saved to /var/cache/conftool/dbconfig/20250110-103657-root.json
* 10:36 marostegui@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P71967 and previous config saved to /var/cache/conftool/dbconfig/20250110-103632-root.json
* 10:36 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P71966 and previous config saved to /var/cache/conftool/dbconfig/20250110-103622-root.json
* 10:31 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2195.codfw.wmnet with reason: host reimage
* 10:30 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1093-1095].eqiad.wmnet
* 10:30 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1093-1095].eqiad.wmnet
* 10:28 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2196.codfw.wmnet with reason: host reimage
* 10:28 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2195.codfw.wmnet with reason: host reimage
* 10:25 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2196.codfw.wmnet with reason: host reimage
* 10:21 marostegui@cumin1002: dbctl commit (dc=all): 'es1022 (re)pooling @ 50%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P71965 and previous config saved to /var/cache/conftool/dbconfig/20250110-102152-root.json
* 10:21 marostegui@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P71964 and previous config saved to /var/cache/conftool/dbconfig/20250110-102126-root.json
* 10:21 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P71963 and previous config saved to /var/cache/conftool/dbconfig/20250110-102116-root.json
* 10:19 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy1022.eqiad.wmnet with reason: maintenance
* 10:19 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dbproxy1022.eqiad.wmnet with reason: maintenance
* 10:07 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2195
* 10:07 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2195
* 10:07 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2195
* 10:07 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2195.codfw.wmnet 225.48.192.10.in-addr.arpa 5.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:07 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2195.codfw.wmnet 225.48.192.10.in-addr.arpa 5.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:07 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:06 marostegui@cumin1002: dbctl commit (dc=all): 'es1022 (re)pooling @ 25%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P71962 and previous config saved to /var/cache/conftool/dbconfig/20250110-100646-root.json
* 10:06 marostegui@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P71961 and previous config saved to /var/cache/conftool/dbconfig/20250110-100621-root.json
* 10:06 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P71960 and previous config saved to /var/cache/conftool/dbconfig/20250110-100611-root.json
* 10:06 elukey: restart dump_cloud_ip_ranges on puppetserver1001 - unit failed due to errors while fetching new data from upstream, trying to see if it was a temporary issue
* 10:05 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2196
* 10:05 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2196
* 10:05 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2196
* 10:05 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2196.codfw.wmnet 224.48.192.10.in-addr.arpa 4.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:05 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 10:05 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2196.codfw.wmnet 224.48.192.10.in-addr.arpa 4.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:05 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:05 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2196 - jelto@cumin1002"
* 10:05 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2196 - jelto@cumin1002"
* 10:03 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db[2128,2186,2228].codfw.wmnet with reason: maintenance
* 10:03 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db[2128,2186,2228].codfw.wmnet with reason: maintenance
* 10:02 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2128 db2228 T373579', diff saved to https://phabricator.wikimedia.org/P71959 and previous config saved to /var/cache/conftool/dbconfig/20250110-100248-marostegui.json
* 10:02 elukey: elukey@cumin1002:~$ sudo cumin -b 20 'an-worker*' 'apt-get clean' (safety to free space and avoid issues on hadoop) - T383320
* 10:01 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 10:00 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2196
* 10:00 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2195
* 10:00 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2196.codfw.wmnet with OS bookworm
* 10:00 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2195.codfw.wmnet with OS bookworm
* 09:57 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2195.codfw.wmnet wikikube-worker2196.codfw.wmnet wikikube-worker2197.codfw.wmnet wikikube-worker2198.codfw.wmnet on all recursors
* 09:57 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2195.codfw.wmnet wikikube-worker2196.codfw.wmnet wikikube-worker2197.codfw.wmnet wikikube-worker2198.codfw.wmnet on all recursors
* 09:57 elukey: kill hanging jupyterhub process on stat1009 to allow puppet to run an delete a user
* 09:56 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2052 to wikikube-worker2198
* 09:56 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2198
* 09:56 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2198
* 09:56 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:56 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2052 to wikikube-worker2198 - jelto@cumin1002"
* 09:55 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2052 to wikikube-worker2198 - jelto@cumin1002"
* 09:54 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db[2132,2160,2232].codfw.wmnet with reason: maintenance
* 09:54 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db[2132,2160,2232].codfw.wmnet with reason: maintenance
* 09:52 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 09:51 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2052 to wikikube-worker2198
* 09:51 marostegui@cumin1002: dbctl commit (dc=all): 'es1022 (re)pooling @ 10%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P71957 and previous config saved to /var/cache/conftool/dbconfig/20250110-095141-root.json
* 09:50 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2051 to wikikube-worker2197
* 09:50 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2197
* 09:50 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2197
* 09:50 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:50 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2051 to wikikube-worker2197 - jelto@cumin1002"
* 09:49 elukey: elukey@cumin1002:~$ sudo cumin 'an-worker11[39,15,54,90,75,57,89,18,06,24]*' 'apt-get clean'
* 09:49 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2051 to wikikube-worker2197 - jelto@cumin1002"
* 09:46 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 09:45 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2051 to wikikube-worker2197
* 09:45 elukey: elukey@cumin1002:~$ sudo cumin 'an-worker11[16,43,19,47,56,72,69]*' 'apt-get clean'
* 09:45 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2050 to wikikube-worker2196
* 09:44 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2196
* 09:44 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2196
* 09:44 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:44 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2050 to wikikube-worker2196 - jelto@cumin1002"
* 09:43 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2050 to wikikube-worker2196 - jelto@cumin1002"
* 09:40 elukey: `apt-get clean` on an-worker1147 to free space on the root partition
* 09:39 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 09:39 elukey: `apt-get clean` on an-worker1117 to free space on the root partition
* 09:39 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2050 to wikikube-worker2196
* 09:38 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2049 to wikikube-worker2195
* 09:38 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2195
* 09:37 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2195
* 09:37 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:37 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2049 to wikikube-worker2195 - jelto@cumin1002"
* 09:36 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2049 to wikikube-worker2195 - jelto@cumin1002"
* 09:33 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 09:32 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2049 to wikikube-worker2195
* 09:28 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes[2049-2052].codfw.wmnet
* 09:25 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes[2049-2052].codfw.wmnet
* 09:21 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2022.codfw.wmnet
* 09:21 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2022.codfw.wmnet
* 09:19 jelto: homer 'lsw1-c6-codfw*' commit 'T377877'
* 09:00 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab to new version
* 08:39 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1243.eqiad.wmnet
* 08:39 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1243.eqiad.wmnet
* 08:36 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1081.eqiad.wmnet
* 08:36 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1081.eqiad.wmnet
* 08:34 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1073.eqiad.wmnet
* 08:34 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1073.eqiad.wmnet
* 08:31 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1069.eqiad.wmnet
* 08:31 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1069.eqiad.wmnet
* 08:25 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1057.eqiad.wmnet
* 08:25 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet
* 08:21 jelto: homer 'lsw1-e3-eqiad*' commit 'T377876'
* 08:15 jelto: homer 'cr*eqiad*' commit 'T377876'
* 05:02 eileen: civicrm upgraded from to 
* 02:36 eileen: civicrm upgraded from to 
* 01:32 eileen: config revision changed from to 
* 01:07 eileen: civicrm upgraded from to 
== 2025-01-09 ==
* 23:00 inflatador: bking@pcc-db1002.puppet-diffs.eqiad1.wikimedia.cloud sudo -u jenkins-deploy /usr/local/sbin/pcc_facts_processor T378368
* 23:00 inflatador: bking@puppetserver1001:~$ sudo /usr/local/sbin/puppet-facts-upload --proxy http://webproxy.eqiad.wmnet:8080 T378368
* 21:44 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Sync cloudelastic1011 status change after Netbox update - bking@cumin2002 - T378368"
* 21:43 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Sync cloudelastic1011 status change after Netbox update - bking@cumin2002 - T378368"
* 20:30 aqu@deploy2002: Finished deploy [airflow-dags/analytics@0e4370e]: Canary event fix (duration: 01m 23s)
* 20:29 aqu@deploy2002: Started deploy [airflow-dags/analytics@0e4370e]: Canary event fix
* 20:06 dcausse@deploy2002: Finished deploy [airflow-dags/search@718e870]: search: switch query_clicks to SparkSqlOperator (duration: 00m 27s)
* 20:05 dcausse@deploy2002: Started deploy [airflow-dags/search@718e870]: search: switch query_clicks to SparkSqlOperator
* 19:54 cdanis@deploy2002: helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: apply
* 19:54 cdanis@deploy2002: helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: apply
* 19:49 cdanis@deploy2002: helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: apply
* 19:34 dduvall@deploy2002: rebuilt and synchronized wikiversions files: group2 to 1.44.0-wmf.11 refs T382362
* 18:49 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-wikifunctions: apply
* 18:46 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-wikifunctions: apply
* 18:46 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-wikifunctions: apply
* 18:45 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-wikifunctions: apply
* 18:45 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply
* 18:44 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1095.eqiad.wmnet with OS bookworm
* 18:44 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
* 18:44 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply
* 18:42 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply
* 18:42 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 18:41 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 18:41 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 18:40 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1094.eqiad.wmnet with OS bookworm
* 18:39 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 18:37 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 18:36 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1093.eqiad.wmnet with OS bookworm
* 18:36 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 18:35 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
* 18:33 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/mw-web: apply
* 18:27 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 18:26 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 18:25 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1095.eqiad.wmnet with reason: host reimage
* 18:24 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
* 18:23 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 18:20 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1094.eqiad.wmnet with reason: host reimage
* 18:17 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1093.eqiad.wmnet with reason: host reimage
* 18:14 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1094.eqiad.wmnet with reason: host reimage
* 18:13 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1095.eqiad.wmnet with reason: host reimage
* 18:13 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1093.eqiad.wmnet with reason: host reimage
* 17:58 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1095
* 17:58 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1095
* 17:58 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1095.eqiad.wmnet with OS bookworm
* 17:58 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1094
* 17:58 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1094
* 17:58 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1094.eqiad.wmnet with OS bookworm
* 17:58 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1093
* 17:58 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1093
* 17:58 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1093.eqiad.wmnet with OS bookworm
* 17:55 kamila@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker1095.eqiad.wmnet with OS bookworm
* 17:55 kamila@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker1094.eqiad.wmnet with OS bookworm
* 17:55 kamila@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker1093.eqiad.wmnet with OS bookworm
* 17:50 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-wikifunctions: apply
* 17:49 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-wikifunctions: apply
* 17:49 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-wikifunctions: apply
* 17:48 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-wikifunctions: apply
* 17:48 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
* 17:47 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 17:47 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
* 17:45 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-web: apply
* 17:45 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 17:44 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 17:44 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 17:42 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 17:42 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 17:41 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 17:41 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 17:41 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2022.codfw.wmnet with OS bookworm
* 17:39 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 17:39 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply
* 17:38 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
* 17:38 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply
* 17:37 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1095
* 17:37 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1095
* 17:37 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1095.eqiad.wmnet with OS bookworm
* 17:37 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply
* 17:37 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1094
* 17:37 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1094
* 17:37 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1094.eqiad.wmnet with OS bookworm
* 17:26 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
* 17:25 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 17:25 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
* 17:22 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-web: apply
* 17:22 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
* 17:21 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
* 17:21 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
* 17:21 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1093
* 17:20 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1093
* 17:20 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1093.eqiad.wmnet with OS bookworm
* 17:20 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2022.codfw.wmnet with reason: host reimage
* 17:19 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
* 17:19 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
* 17:18 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
* 17:18 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
* 17:17 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
* 17:17 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-wikifunctions: apply
* 17:16 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2022.codfw.wmnet with reason: host reimage
* 17:16 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-wikifunctions: apply
* 17:16 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-wikifunctions: apply
* 17:15 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-wikifunctions: apply
* 17:02 cdanis@deploy2002: Finished scap sync-world: Backport for gerrit:1109133group1: enable OpenTelemetry exports (T340552) (duration: 14m 22s)
* 16:55 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
* 16:55 elukey@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
* 16:55 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2022.codfw.wmnet with OS bookworm
* 16:54 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
* 16:54 elukey@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
* 16:54 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
* 16:54 elukey@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2022
* 16:54 elukey@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2022
* 16:54 elukey@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2022.codfw.wmnet 212.32.192.10.in-addr.arpa 2.1.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 16:54 elukey@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2022.codfw.wmnet 212.32.192.10.in-addr.arpa 2.1.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 16:54 elukey@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:54 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2022 - elukey@cumin1002"
* 16:54 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2022 - elukey@cumin1002"
* 16:53 cdanis@deploy2002: cdanis: Continuing with sync
* 16:53 cdanis@deploy2002: cdanis: Backport for gerrit:1109133group1: enable OpenTelemetry exports (T340552) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 16:50 elukey@cumin1002: START - Cookbook sre.dns.netbox
* 16:50 elukey@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
* 16:48 cdanis@deploy2002: Started scap sync-world: Backport for gerrit:1109133group1: enable OpenTelemetry exports (T340552)
* 16:40 dduvall@deploy2002: rebuilt and synchronized wikiversions files: group1 to 1.44.0-wmf.11 refs T382362
* 16:33 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1243.eqiad.wmnet with OS bookworm
* 16:33 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
* 16:29 bking@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
* 16:28 bking@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
* 16:27 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
* 16:21 kamila@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1095.eqiad.wmnet with OS bookworm
* 16:21 kamila@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1094.eqiad.wmnet with OS bookworm
* 16:21 kamila@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1093.eqiad.wmnet with OS bookworm
* 16:21 bking@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
* 16:21 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1069.eqiad.wmnet with OS bookworm
* 16:21 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
* 16:20 bking@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
* 16:12 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
* 16:08 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1243.eqiad.wmnet with reason: host reimage
* 16:05 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1243.eqiad.wmnet with reason: host reimage
* 15:56 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1057.eqiad.wmnet with OS bookworm
* 15:56 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
* 15:55 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
* 15:54 sukhe@dns1004:: END - running authdns-update
* 15:53 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1069.eqiad.wmnet with reason: host reimage
* 15:52 bking@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
* 15:52 sukhe@dns1004:: START - running authdns-update
* 15:52 bking@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
* 15:49 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1069.eqiad.wmnet with reason: host reimage
* 15:46 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1243.eqiad.wmnet with OS bookworm
* 15:46 inflatador: bking@an-airflow1005 stopping airflow-search services as part of k8s migration T380615
* 15:46 tchin@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 15:45 tchin@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 15:39 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1073.eqiad.wmnet with OS bookworm
* 15:39 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
* 15:39 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
* 15:37 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1057.eqiad.wmnet with reason: host reimage
* 15:33 tchin@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 15:33 tchin@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 15:33 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1057.eqiad.wmnet with reason: host reimage
* 15:31 sukhe@dns1004:: END - running authdns-update
* 15:31 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1069.eqiad.wmnet with OS bookworm
* 15:30 sukhe@dns1004:: START - running authdns-update
* 15:28 logmsgbot: testing update from dns host
* 15:27 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2193-2194].codfw.wmnet
* 15:27 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2193-2194].codfw.wmnet
* 15:25 jelto: homer 'cr*codfw*' commit 'T377877'
* 15:24 jelto: homer 'lsw1-c5-codfw*' commit 'T377877'
* 15:23 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1095
* 15:23 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1095
* 15:23 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1095.eqiad.wmnet with OS bookworm
* 15:23 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1094
* 15:23 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1094
* 15:23 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1094.eqiad.wmnet with OS bookworm
* 15:23 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1093
* 15:23 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1093
* 15:23 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1093.eqiad.wmnet with OS bookworm
* 15:22 tchin@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 15:22 tchin@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 15:21 jelto: homer 'lsw1-d3-codfw*' commit 'T377877'
* 15:21 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1093.eqiad.wmnet wikikube-worker1094.eqiad.wmnet wikikube-worker1095.eqiad.wmnet on all recursors
* 15:21 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1093.eqiad.wmnet wikikube-worker1094.eqiad.wmnet wikikube-worker1095.eqiad.wmnet on all recursors
* 15:21 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1457 to wikikube-worker1093
* 15:20 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1093
* 15:20 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1073.eqiad.wmnet with reason: host reimage
* 15:19 jelto@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker2192.codfw.wmnet with OS bookworm
* 15:19 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1093
* 15:18 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:18 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1459 to wikikube-worker1095
* 15:17 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1095
* 15:16 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1073.eqiad.wmnet with reason: host reimage
* 15:16 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1458 to wikikube-worker1094
* 15:16 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 15:16 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1095
* 15:16 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:15 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1094
* 15:15 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1057.eqiad.wmnet with OS bookworm
* 15:14 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1094
* 15:14 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:14 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1458 to wikikube-worker1094 - kamila@cumin1002"
* 15:14 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 15:12 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1458 to wikikube-worker1094 - kamila@cumin1002"
* 15:09 tchin@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 15:09 tchin@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
* 15:09 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 15:09 kamila@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 15:08 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1459 to wikikube-worker1095
* 15:07 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1458 to wikikube-worker1094
* 15:06 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 15:06 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1457 to wikikube-worker1093
* 15:06 Lucas_WMDE: UTC afternoon backport+config window done
* 15:04 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1099725Enable Translate message bundle Scribunto library on MetaWiki (T379892) (duration: 32m 53s)
* 14:58 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1073.eqiad.wmnet with OS bookworm
* 14:56 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2194.codfw.wmnet with OS bookworm
* 14:54 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[1457-1459].eqiad.wmnet
* 14:54 lucaswerkmeister-wmde@deploy2002: abi, lucaswerkmeister-wmde: Continuing with sync
* 14:52 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2193.codfw.wmnet with OS bookworm
* 14:52 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[1457-1459].eqiad.wmnet
* 14:41 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2192
* 14:41 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2192
* 14:41 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2192.codfw.wmnet with OS bookworm
* 14:38 lucaswerkmeister-wmde@deploy2002: abi, lucaswerkmeister-wmde: Backport for gerrit:1099725Enable Translate message bundle Scribunto library on MetaWiki (T379892) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:37 dcaro@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 14:37 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2194.codfw.wmnet with reason: host reimage
* 14:33 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2193.codfw.wmnet with reason: host reimage
* 14:32 dcaro@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 14:31 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1099725Enable Translate message bundle Scribunto library on MetaWiki (T379892)
* 14:31 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2194.codfw.wmnet with reason: host reimage
* 14:29 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2193.codfw.wmnet with reason: host reimage
* 14:24 dcaro@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 14:12 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2194
* 14:12 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2194
* 14:12 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2194.codfw.wmnet with OS bookworm
* 14:11 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2193
* 14:11 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2193
* 14:11 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2193.codfw.wmnet with OS bookworm
* 14:05 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2194.codfw.wmnet with OS bookworm
* 14:00 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2193.codfw.wmnet with OS bookworm
* 13:58 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2192.codfw.wmnet with OS bookworm
* 13:39 moritzm: installing jinja2 security updates
* 13:21 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1088-1092].eqiad.wmnet
* 13:21 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1088-1092].eqiad.wmnet
* 13:21 aqu@deploy2002: Finished deploy [airflow-dags/analytics@9073e46]: Refine refactoring (duration: 02m 51s)
* 13:18 aqu@deploy2002: Started deploy [airflow-dags/analytics@9073e46]: Refine refactoring
* 13:11 moritzm: installing sqlparse security updates
* 13:08 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 100%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71948 and previous config saved to /var/cache/conftool/dbconfig/20250109-130818-root.json
* 12:54 aqu@deploy2002: deploy aborted: Refine refactoring (duration: 00m 20s)
* 12:54 aqu@deploy2002: Started deploy [airflow-dags/analytics_test@9073e46]: Refine refactoring
* 12:53 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 75%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71947 and previous config saved to /var/cache/conftool/dbconfig/20250109-125313-root.json
* 12:45 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2194
* 12:45 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2194
* 12:45 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2194
* 12:45 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2194.codfw.wmnet 224.32.192.10.in-addr.arpa 4.2.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 12:45 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2194.codfw.wmnet 224.32.192.10.in-addr.arpa 4.2.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 12:45 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:45 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2194 - jelto@cumin1002"
* 12:44 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2194 - jelto@cumin1002"
* 12:43 dcaro@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bullseye
* 12:41 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2193
* 12:41 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2193
* 12:41 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 12:41 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2193
* 12:41 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2193.codfw.wmnet 62.48.192.10.in-addr.arpa 2.6.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 12:41 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2193.codfw.wmnet 62.48.192.10.in-addr.arpa 2.6.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 12:41 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:41 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2193 - jelto@cumin1002"
* 12:41 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2193 - jelto@cumin1002"
* 12:40 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2194
* 12:40 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2194.codfw.wmnet with OS bookworm
* 12:38 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 50%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71946 and previous config saved to /var/cache/conftool/dbconfig/20250109-123806-root.json
* 12:38 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2192
* 12:38 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2192
* 12:37 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2192
* 12:37 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2192.codfw.wmnet 221.48.192.10.in-addr.arpa 1.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 12:37 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 12:37 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2192.codfw.wmnet 221.48.192.10.in-addr.arpa 1.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 12:37 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:37 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2192 - jelto@cumin1002"
* 12:37 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2192 - jelto@cumin1002"
* 12:37 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2193
* 12:37 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2193.codfw.wmnet with OS bookworm
* 12:34 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 12:34 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2192
* 12:34 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2192.codfw.wmnet with OS bookworm
* 12:31 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2192.codfw.wmnet wikikube-worker2193.codfw.wmnet wikikube-worker2194.codfw.wmnet on all recursors
* 12:31 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2192.codfw.wmnet wikikube-worker2193.codfw.wmnet wikikube-worker2194.codfw.wmnet on all recursors
* 12:30 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2058 to wikikube-worker2194
* 12:30 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2194
* 12:30 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2194
* 12:30 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:30 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2058 to wikikube-worker2194 - jelto@cumin1002"
* 12:29 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2058 to wikikube-worker2194 - jelto@cumin1002"
* 12:26 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 12:25 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2058 to wikikube-worker2194
* 12:25 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2056 to wikikube-worker2193
* 12:25 root@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Piccardi out of all services on: 2310 hosts
* 12:24 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2193
* 12:24 root@cumin2002: START - Cookbook sre.idm.logout Logging Piccardi out of all services on: 2310 hosts
* 12:23 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 25%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71945 and previous config saved to /var/cache/conftool/dbconfig/20250109-122301-root.json
* 12:19 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2193
* 12:19 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:19 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2056 to wikikube-worker2193 - jelto@cumin1002"
* 12:18 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2056 to wikikube-worker2193 - jelto@cumin1002"
* 12:15 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 12:14 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2056 to wikikube-worker2193
* 12:13 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2053 to wikikube-worker2192
* 12:13 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2192
* 12:13 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2192
* 12:12 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:12 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2053 to wikikube-worker2192 - jelto@cumin1002"
* 12:12 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2053 to wikikube-worker2192 - jelto@cumin1002"
* 12:08 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 12:07 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 10%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71944 and previous config saved to /var/cache/conftool/dbconfig/20250109-120755-root.json
* 12:07 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 12:06 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 12:05 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 12:05 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 12:05 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2053 to wikikube-worker2192
* 12:04 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 12:03 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 11:58 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 11:57 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 11:54 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 11:53 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 11:53 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 11:53 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 11:52 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 9%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71943 and previous config saved to /var/cache/conftool/dbconfig/20250109-115250-root.json
* 11:37 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 8%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71942 and previous config saved to /var/cache/conftool/dbconfig/20250109-113744-root.json
* 11:29 elukey@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker2022.codfw.wmnet with OS bookworm
* 11:25 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 100%: Repooling after moving sanitarium', diff saved to https://phabricator.wikimedia.org/P71940 and previous config saved to /var/cache/conftool/dbconfig/20250109-112543-root.json
* 11:25 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 100%: Repooling after moving sanitarium', diff saved to https://phabricator.wikimedia.org/P71939 and previous config saved to /var/cache/conftool/dbconfig/20250109-112534-root.json
* 11:24 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
* 11:24 elukey@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
* 11:23 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2022.codfw.wmnet with OS bookworm
* 11:22 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 7%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71938 and previous config saved to /var/cache/conftool/dbconfig/20250109-112239-root.json
* 11:10 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 75%: Repooling after moving sanitarium', diff saved to https://phabricator.wikimedia.org/P71937 and previous config saved to /var/cache/conftool/dbconfig/20250109-111038-root.json
* 11:10 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 75%: Repooling after moving sanitarium', diff saved to https://phabricator.wikimedia.org/P71936 and previous config saved to /var/cache/conftool/dbconfig/20250109-111029-root.json
* 11:07 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 6%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71935 and previous config saved to /var/cache/conftool/dbconfig/20250109-110734-root.json
* 10:57 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4', diff saved to https://phabricator.wikimedia.org/P71934 and previous config saved to /var/cache/conftool/dbconfig/20250109-105708-ladsgroup.json
* 10:55 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 50%: Repooling after moving sanitarium', diff saved to https://phabricator.wikimedia.org/P71933 and previous config saved to /var/cache/conftool/dbconfig/20250109-105533-root.json
* 10:55 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 50%: Repooling after moving sanitarium', diff saved to https://phabricator.wikimedia.org/P71932 and previous config saved to /var/cache/conftool/dbconfig/20250109-105523-root.json
* 10:52 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 5%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71931 and previous config saved to /var/cache/conftool/dbconfig/20250109-105228-root.json
* 10:45 ladsgroup@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for pc2015.codfw.wmnet
* 10:45 root@cumin2002: END (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for dbprov2004.codfw.wmnet: Renew puppet certificate - root@cumin2002
* 10:45 root@cumin2002: START - Cookbook sre.puppet.renew-cert for dbprov2004.codfw.wmnet: Renew puppet certificate - root@cumin2002
* 10:43 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc1016.eqiad.wmnet with reason: Reboot
* 10:43 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on pc1016.eqiad.wmnet with reason: Reboot
* 10:40 marostegui@cumin1002: dbctl commit (dc=all): 'db2131 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P71930 and previous config saved to /var/cache/conftool/dbconfig/20250109-104043-root.json
* 10:40 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 25%: Repooling after moving sanitarium', diff saved to https://phabricator.wikimedia.org/P71929 and previous config saved to /var/cache/conftool/dbconfig/20250109-104027-root.json
* 10:40 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 25%: Repooling after moving sanitarium', diff saved to https://phabricator.wikimedia.org/P71928 and previous config saved to /var/cache/conftool/dbconfig/20250109-104017-root.json
* 10:40 ladsgroup@cumin1002: START - Cookbook sre.mysql.upgrade for pc2015.codfw.wmnet
* 10:38 ladsgroup@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for pc1016.eqiad.wmnet
* 10:37 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 4%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71927 and previous config saved to /var/cache/conftool/dbconfig/20250109-103723-root.json
* 10:34 ladsgroup@cumin1002: START - Cookbook sre.mysql.upgrade for pc1016.eqiad.wmnet
* 10:27 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling pc4', diff saved to https://phabricator.wikimedia.org/P71926 and previous config saved to /var/cache/conftool/dbconfig/20250109-102700-ladsgroup.json
* 10:25 marostegui@cumin1002: dbctl commit (dc=all): 'db2131 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P71925 and previous config saved to /var/cache/conftool/dbconfig/20250109-102538-root.json
* 10:25 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 10%: Repooling after moving sanitarium', diff saved to https://phabricator.wikimedia.org/P71924 and previous config saved to /var/cache/conftool/dbconfig/20250109-102522-root.json
* 10:25 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 10%: Repooling after moving sanitarium', diff saved to https://phabricator.wikimedia.org/P71923 and previous config saved to /var/cache/conftool/dbconfig/20250109-102512-root.json
* 10:22 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 3%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71922 and previous config saved to /var/cache/conftool/dbconfig/20250109-102218-root.json
* 10:21 marostegui: Move db2187:3312 under db2226 s2 codfw dbmaint T373579
* 10:20 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db[2126,2187,2226].codfw.wmnet with reason: maintenance
* 10:20 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db[2126,2187,2226].codfw.wmnet with reason: maintenance
* 10:20 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2126 db2226 T373579', diff saved to https://phabricator.wikimedia.org/P71921 and previous config saved to /var/cache/conftool/dbconfig/20250109-102010-marostegui.json
* 10:12 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes[2053,2056,2058].codfw.wmnet
* 10:11 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes[2053,2056,2058].codfw.wmnet
* 10:10 marostegui@cumin1002: dbctl commit (dc=all): 'db2131 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P71920 and previous config saved to /var/cache/conftool/dbconfig/20250109-101033-root.json
* 10:09 klausman@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2001.codfw.wmnet
* 10:07 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 2%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71919 and previous config saved to /var/cache/conftool/dbconfig/20250109-100712-root.json
* 10:06 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbprov2004.codfw.wmnet with reason: os upgrade
* 10:05 jynus@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbprov2004.codfw.wmnet with reason: os upgrade
* 09:58 moritzm: installing glibc bugfix updates for Bookworm
* 09:55 klausman@cumin2002: START - Cookbook sre.hosts.reboot-single for host ml-serve2001.codfw.wmnet
* 09:55 marostegui@cumin1002: dbctl commit (dc=all): 'db2131 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P71918 and previous config saved to /var/cache/conftool/dbconfig/20250109-095527-root.json
* 09:55 klausman@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2001.codfw.wmnet
* 09:52 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 1%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71917 and previous config saved to /var/cache/conftool/dbconfig/20250109-095207-root.json
* 09:43 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 100%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71916 and previous config saved to /var/cache/conftool/dbconfig/20250109-094355-root.json
* 09:40 marostegui@cumin1002: dbctl commit (dc=all): 'db2131 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P71915 and previous config saved to /var/cache/conftool/dbconfig/20250109-094022-root.json
* 09:28 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 75%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71914 and previous config saved to /var/cache/conftool/dbconfig/20250109-092850-root.json
* 09:23 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp4044.ulsfo.wmnet,cp4052.ulsfo.wmnet} and A:cp
* 09:21 klausman@cumin2002: START - Cookbook sre.hosts.reboot-single for host ml-serve2001.codfw.wmnet
* 09:20 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2013.codfw.wmnet
* 09:19 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet
* 09:19 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2012.codfw.wmnet
* 09:19 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2012.codfw.wmnet
* 09:19 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2011.codfw.wmnet
* 09:19 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet
* 09:19 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2013.codfw.wmnet with OS bookworm
* 09:18 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp4044.ulsfo.wmnet,cp4052.ulsfo.wmnet} and A:cp
* 09:17 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp4044.ulsfo.wmnet,cp4052.ulsfo.wmnet} and A:cp
* 09:15 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2012.codfw.wmnet with OS bookworm
* 09:13 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 50%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71913 and previous config saved to /var/cache/conftool/dbconfig/20250109-091345-root.json
* 09:13 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp4044.ulsfo.wmnet,cp4052.ulsfo.wmnet} and A:cp
* 09:12 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2011.codfw.wmnet with OS bookworm
* 09:02 vgutierrez: update to haproxy 2.8.13 on component thirdparty/haproxy28 bullseye-wikimedia (apt.wm.o) - T383111
* 08:59 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2013.codfw.wmnet with reason: host reimage
* 08:58 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 25%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71912 and previous config saved to /var/cache/conftool/dbconfig/20250109-085840-root.json
* 08:57 root@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2131.codfw.wmnet onto db2231.codfw.wmnet
* 08:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2012.codfw.wmnet with reason: host reimage
* 08:52 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2011.codfw.wmnet with reason: host reimage
* 08:49 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2013.codfw.wmnet with reason: host reimage
* 08:49 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2012.codfw.wmnet with reason: host reimage
* 08:49 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2011.codfw.wmnet with reason: host reimage
* 08:43 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 10%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71911 and previous config saved to /var/cache/conftool/dbconfig/20250109-084335-root.json
* 08:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es1043.eqiad.wmnet with reason: cloning
* 08:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on es1043.eqiad.wmnet with reason: cloning
* 08:32 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2012
* 08:32 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2012
* 08:32 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2013
* 08:32 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2013
* 08:31 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2012.codfw.wmnet with OS bookworm
* 08:31 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2013.codfw.wmnet with OS bookworm
* 08:31 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2011
* 08:31 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2011
* 08:31 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2011.codfw.wmnet with OS bookworm
* 08:28 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 9%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71910 and previous config saved to /var/cache/conftool/dbconfig/20250109-082829-root.json
* 08:26 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2011-2013].codfw.wmnet
* 08:24 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2011-2013].codfw.wmnet
* 08:22 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2014.codfw.wmnet
* 08:22 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2014.codfw.wmnet
* 08:22 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2017.codfw.wmnet
* 08:22 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2017.codfw.wmnet
* 08:21 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2017.codfw.wmnet with OS bookworm
* 08:15 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2014.codfw.wmnet with OS bookworm
* 08:13 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 8%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71909 and previous config saved to /var/cache/conftool/dbconfig/20250109-081324-root.json
* 08:01 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2017.codfw.wmnet with reason: host reimage
* 07:58 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 7%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71908 and previous config saved to /var/cache/conftool/dbconfig/20250109-075820-root.json
* 07:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2014.codfw.wmnet with reason: host reimage
* 07:52 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2017.codfw.wmnet with reason: host reimage
* 07:52 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2014.codfw.wmnet with reason: host reimage
* 07:43 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 6%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71907 and previous config saved to /var/cache/conftool/dbconfig/20250109-074314-root.json
* 07:42 marostegui@cumin1002: dbctl commit (dc=all): 'es1021 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P71906 and previous config saved to /var/cache/conftool/dbconfig/20250109-074214-root.json
* 07:34 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2017
* 07:34 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2017
* 07:34 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2014
* 07:34 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2014
* 07:34 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2017.codfw.wmnet with OS bookworm
* 07:34 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2014.codfw.wmnet with OS bookworm
* 07:32 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2014,2017].codfw.wmnet
* 07:29 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2014,2017].codfw.wmnet
* 07:28 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 5%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71905 and previous config saved to /var/cache/conftool/dbconfig/20250109-072809-root.json
* 07:27 marostegui@cumin1002: dbctl commit (dc=all): 'es1021 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P71904 and previous config saved to /var/cache/conftool/dbconfig/20250109-072709-root.json
* 07:13 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 4%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71903 and previous config saved to /var/cache/conftool/dbconfig/20250109-071305-root.json
* 07:12 marostegui@cumin1002: dbctl commit (dc=all): 'es1021 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P71902 and previous config saved to /var/cache/conftool/dbconfig/20250109-071203-root.json
* 06:58 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 3%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71901 and previous config saved to /var/cache/conftool/dbconfig/20250109-065759-root.json
* 06:56 marostegui@cumin1002: dbctl commit (dc=all): 'es1021 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P71900 and previous config saved to /var/cache/conftool/dbconfig/20250109-065658-root.json
* 06:53 kart_: Updated cxserver to 2025-01-07-045930-production (T377966, T377813, T381379)
* 06:52 root@cumin1002: START - Cookbook sre.mysql.clone of db2131.codfw.wmnet onto db2231.codfw.wmnet
* 06:51 marostegui@cumin1002: dbctl commit (dc=all): 'Add db2231 to dbctl depooled T373579', diff saved to https://phabricator.wikimedia.org/P71899 and previous config saved to /var/cache/conftool/dbconfig/20250109-065114-marostegui.json
* 06:47 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2131.codfw.wmnet with reason: cloning db2231
* 06:47 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2131.codfw.wmnet with reason: cloning db2231
* 06:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2131 T373579', diff saved to https://phabricator.wikimedia.org/P71898 and previous config saved to /var/cache/conftool/dbconfig/20250109-064556-marostegui.json
* 06:45 kartik@deploy2002: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
* 06:44 kartik@deploy2002: helmfile [eqiad] START helmfile.d/services/cxserver: apply
* 06:42 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 2%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71897 and previous config saved to /var/cache/conftool/dbconfig/20250109-064254-root.json
* 06:41 marostegui@cumin1002: dbctl commit (dc=all): 'es1021 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P71896 and previous config saved to /var/cache/conftool/dbconfig/20250109-064153-root.json
* 06:41 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es1022.eqiad.wmnet with reason: cloning es1042
* 06:41 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on es1022.eqiad.wmnet with reason: cloning es1042
* 06:41 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1022 T382569', diff saved to https://phabricator.wikimedia.org/P71895 and previous config saved to /var/cache/conftool/dbconfig/20250109-064117-marostegui.json
* 06:40 kartik@deploy2002: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
* 06:40 kartik@deploy2002: helmfile [codfw] START helmfile.d/services/cxserver: apply
* 06:31 kartik@deploy2002: helmfile [staging] DONE helmfile.d/services/cxserver: apply
* 06:30 kartik@deploy2002: helmfile [staging] START helmfile.d/services/cxserver: apply
* 06:27 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 1%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71894 and previous config saved to /var/cache/conftool/dbconfig/20250109-062749-root.json
* 06:26 marostegui@cumin1002: dbctl commit (dc=all): 'es1021 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P71893 and previous config saved to /var/cache/conftool/dbconfig/20250109-062647-root.json
* 06:17 marostegui@cumin1002: dbctl commit (dc=all): 'Add es1042 depooled T382569', diff saved to https://phabricator.wikimedia.org/P71892 and previous config saved to /var/cache/conftool/dbconfig/20250109-061724-marostegui.json
* 06:11 marostegui@cumin1002: dbctl commit (dc=all): 'es1021 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P71891 and previous config saved to /var/cache/conftool/dbconfig/20250109-061142-root.json
* 00:43 ladsgroup@deploy2002: Finished scap sync-world: Backport for gerrit:1109149filerepo: Fix schema compatibility constant usage (T383269) (duration: 13m 34s)
* 00:36 ladsgroup@deploy2002: ladsgroup, cdanis: Continuing with sync
* 00:36 ladsgroup@deploy2002: ladsgroup, cdanis: Backport for gerrit:1109149filerepo: Fix schema compatibility constant usage (T383269) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 00:30 ladsgroup@deploy2002: Started scap sync-world: Backport for gerrit:1109149filerepo: Fix schema compatibility constant usage (T383269)
== 2025-01-08 ==
* 23:03 eileen: civicrm upgraded from to 
* 22:27 eileen: civicrm upgraded from to 
* 21:28 dzahn@cumin2002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: security release 
* 21:16 aokoth@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Security Update
* 21:14 dzahn@cumin2002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: security release 
* 21:08 aokoth@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Update
* 20:05 dduvall@deploy2002: rebuilt and synchronized wikiversions files: group0 to 1.44.0-wmf.11 refs T382362
* 19:59 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wikikube-worker1001.eqiad.wmnet
* 19:59 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 19:59 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wikikube-worker1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - kamila@cumin1002"
* 19:55 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wikikube-worker1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - kamila@cumin1002"
* 19:54 dduvall: rolling back wmf.11 to group0 due to `Table 'commonswiki.file' doesn't exist` errors
* 19:53 kamila_: homer 'cr*eqiad*' commit 'T365571'
* 19:52 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 19:46 kamila@cumin1002: START - Cookbook sre.hosts.decommission for hosts wikikube-worker1001.eqiad.wmnet
* 19:32 sfaci@deploy2002: Finished deploy [airflow-dags/analytics@b2b5707]: (no justification provided) (duration: 03m 06s)
* 19:29 sfaci@deploy2002: Started deploy [airflow-dags/analytics@b2b5707]: (no justification provided)
* 19:29 dduvall@deploy2002: rebuilt and synchronized wikiversions files: group1 to 1.44.0-wmf.11 refs T382362
* 19:17 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1001.eqiad.wmnet
* 19:17 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1001.eqiad.wmnet
* 19:08 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1089.eqiad.wmnet with OS bookworm
* 19:03 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1091.eqiad.wmnet with OS bookworm
* 18:59 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1090.eqiad.wmnet with OS bookworm
* 18:55 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1092.eqiad.wmnet with OS bookworm
* 18:52 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1088.eqiad.wmnet with OS bookworm
* 18:48 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1089.eqiad.wmnet with reason: host reimage
* 18:43 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1091.eqiad.wmnet with reason: host reimage
* 18:40 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1090.eqiad.wmnet with reason: host reimage
* 18:36 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1092.eqiad.wmnet with reason: host reimage
* 18:33 swfrench@deploy2002: Finished scap sync-world: Deployment to switch migration release files to 8.1 - T377040 (duration: 13m 57s)
* 18:32 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1088.eqiad.wmnet with reason: host reimage
* 18:29 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1092.eqiad.wmnet with reason: host reimage
* 18:29 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1091.eqiad.wmnet with reason: host reimage
* 18:29 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1090.eqiad.wmnet with reason: host reimage
* 18:28 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1089.eqiad.wmnet with reason: host reimage
* 18:27 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1088.eqiad.wmnet with reason: host reimage
* 18:19 swfrench@deploy2002: Started scap sync-world: Deployment to switch migration release files to 8.1 - T377040
* 18:17 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcontrol1011.eqiad.wmnet with OS bookworm
* 18:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
* 18:16 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
* 18:13 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1092
* 18:13 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1092
* 18:13 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1092.eqiad.wmnet with OS bookworm
* 18:13 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1091
* 18:13 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1091
* 18:13 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1091.eqiad.wmnet with OS bookworm
* 18:12 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1090
* 18:12 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1090
* 18:12 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1090.eqiad.wmnet with OS bookworm
* 18:12 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1089
* 18:12 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1089
* 18:12 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1089.eqiad.wmnet with OS bookworm
* 18:11 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1088
* 18:11 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1088
* 18:11 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1088.eqiad.wmnet with OS bookworm
* 18:09 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1088.eqiad.wmnet wikikube-worker1089.eqiad.wmnet wikikube-worker1090.eqiad.wmnet wikikube-worker1091.eqiad.wmnet wikikube-worker1092.eqiad.wmnet on all recursors
* 18:09 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1088.eqiad.wmnet wikikube-worker1089.eqiad.wmnet wikikube-worker1090.eqiad.wmnet wikikube-worker1091.eqiad.wmnet wikikube-worker1092.eqiad.wmnet on all recursors
* 18:09 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1455 to wikikube-worker1092
* 18:08 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1092
* 18:07 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1092
* 18:07 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:07 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1455 to wikikube-worker1092 - kamila@cumin1002"
* 18:07 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1455 to wikikube-worker1092 - kamila@cumin1002"
* 18:04 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1454 to wikikube-worker1091
* 18:04 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1091
* 18:03 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 18:02 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1091
* 18:02 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:02 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1454 to wikikube-worker1091 - kamila@cumin1002"
* 18:02 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1454 to wikikube-worker1091 - kamila@cumin1002"
* 18:01 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1455 to wikikube-worker1092
* 17:59 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 17:58 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1454 to wikikube-worker1091
* 17:58 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1088.eqiad.wmnet wikikube-worker1089.eqiad.wmnet wikikube-worker1090.eqiad.wmnet wikikub on all recursors
* 17:58 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1088.eqiad.wmnet wikikube-worker1089.eqiad.wmnet wikikube-worker1090.eqiad.wmnet wikikub on all recursors
* 17:57 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcontrol1011.eqiad.wmnet with reason: host reimage
* 17:56 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1452 to wikikube-worker1089
* 17:56 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1089
* 17:55 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1089
* 17:55 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:54 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcontrol1011.eqiad.wmnet with reason: host reimage
* 17:53 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1453 to wikikube-worker1090
* 17:52 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1090
* 17:52 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 17:51 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1090
* 17:51 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:51 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1453 to wikikube-worker1090 - kamila@cumin1002"
* 17:51 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1453 to wikikube-worker1090 - kamila@cumin1002"
* 17:49 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1451 to wikikube-worker1088
* 17:48 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1088
* 17:47 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 17:47 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1088
* 17:47 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:47 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1451 to wikikube-worker1088 - kamila@cumin1002"
* 17:46 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1453 to wikikube-worker1090
* 17:46 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1451 to wikikube-worker1088 - kamila@cumin1002"
* 17:44 kamila@cumin1002: END (FAIL) - Cookbook sre.hosts.rename (exit_code=99) from mw1453 to wikikube-worker1090
* 17:43 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1453 to wikikube-worker1090
* 17:43 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1452 to wikikube-worker1089
* 17:43 kamila@cumin1002: START - Cookbook sre.dns.netbox
* 17:42 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1451 to wikikube-worker1088
* 17:41 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[1451-1455].eqiad.wmnet
* 17:36 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[1451-1455].eqiad.wmnet
* 17:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 17:05 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 17:04 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2019.codfw.wmnet
* 17:04 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2019.codfw.wmnet
* 17:02 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 17:02 jelto: sudo homer 'cr*codfw*' commit 'T377877'
* 17:01 jelto: sudo homer 'lsw1-c3-codfw*' commit 'T377877'
* 17:00 jelto@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2019.codfw.wmnet
* 17:00 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcontrol1011.eqiad.wmnet with OS bookworm
* 16:59 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2019.codfw.wmnet
* 16:59 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2018.codfw.wmnet
* 16:59 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2018.codfw.wmnet
* 16:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2019.codfw.wmnet with OS bookworm
* 16:50 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1081.eqiad.wmnet with OS bookworm
* 16:50 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
* 16:50 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
* 16:44 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2018.codfw.wmnet with OS bookworm
* 16:35 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2019.codfw.wmnet with reason: host reimage
* 16:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1081.eqiad.wmnet with reason: host reimage
* 16:29 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:27 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
* 16:26 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2019.codfw.wmnet with reason: host reimage
* 16:25 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1081.eqiad.wmnet with reason: host reimage
* 16:24 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2018.codfw.wmnet with reason: host reimage
* 16:20 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:20 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2018.codfw.wmnet with reason: host reimage
* 16:08 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 16:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1081.eqiad.wmnet with OS bookworm
* 16:07 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2019
* 16:07 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2019
* 16:07 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2019
* 16:07 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2019.codfw.wmnet 117.32.192.10.in-addr.arpa 7.1.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 16:07 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2019.codfw.wmnet 117.32.192.10.in-addr.arpa 7.1.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 16:07 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:07 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2019 - jelto@cumin1002"
* 16:07 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2019 - jelto@cumin1002"
* 16:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 16:04 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 16:03 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2019
* 16:03 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2018
* 16:03 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2018
* 16:03 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2019.codfw.wmnet with OS bookworm
* 16:03 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2018.codfw.wmnet with OS bookworm
* 16:02 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2018-2019].codfw.wmnet
* 16:01 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 100%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71886 and previous config saved to /var/cache/conftool/dbconfig/20250108-160158-root.json
* 16:00 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2018-2019].codfw.wmnet
* 15:57 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply
* 15:57 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply
* 15:57 jelto@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker2022.codfw.wmnet with OS bookworm
* 15:54 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
* 15:54 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
* 15:51 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
* 15:51 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
* 15:51 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2022.codfw.wmnet with OS bookworm
* 15:51 jelto@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker2022.codfw.wmnet with OS bookworm
* 15:49 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
* 15:48 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
* 15:46 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 75%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71885 and previous config saved to /var/cache/conftool/dbconfig/20250108-154653-root.json
* 15:39 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
* 15:36 moritzm: installing jinja2 security updates
* 15:31 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 50%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71884 and previous config saved to /var/cache/conftool/dbconfig/20250108-153147-root.json
* 15:29 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
* 15:27 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
* 15:27 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
* 15:22 gengh@deploy2002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 15:22 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
* 15:22 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
* 15:22 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2022.codfw.wmnet with OS bookworm
* 15:21 gengh@deploy2002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 15:21 gengh@deploy2002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 15:20 gengh@deploy2002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 15:18 gengh@deploy2002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 15:17 gengh@deploy2002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 15:16 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 25%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71883 and previous config saved to /var/cache/conftool/dbconfig/20250108-151642-root.json
* 15:12 gengh@deploy2002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 15:11 gengh@deploy2002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 15:11 jelto@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker2022.codfw.wmnet with OS bookworm
* 15:10 gengh@deploy2002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 15:10 gengh@deploy2002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 15:08 gengh@deploy2002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 15:07 gengh@deploy2002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 15:07 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
* 15:07 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
* 15:06 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
* 15:06 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
* 15:06 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2022.codfw.wmnet with OS bookworm
* 15:01 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 10%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71882 and previous config saved to /var/cache/conftool/dbconfig/20250108-150136-root.json
* 15:00 joelyrookewmde: Finished populateSitesTable for tigwiki (T381382)
* 14:53 elukey@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2022.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 14:48 jelto@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker2022.codfw.wmnet with OS bookworm
* 14:48 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc5 (T373037)', diff saved to https://phabricator.wikimedia.org/P71881 and previous config saved to /var/cache/conftool/dbconfig/20250108-144805-ladsgroup.json
* 14:46 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 5%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71880 and previous config saved to /var/cache/conftool/dbconfig/20250108-144631-root.json
* 14:44 marostegui@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 100%: Repooling after cloning', diff saved to https://phabricator.wikimedia.org/P71879 and previous config saved to /var/cache/conftool/dbconfig/20250108-144429-root.json
* 14:42 ladsgroup@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for pc1014.eqiad.wmnet
* 14:40 elukey: elukey@puppetserver1001:~$ sudo puppetserver ca clean --certname kubernetes1061.eqiad.wmnet
* 14:39 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
* 14:38 joelyrookewmde: joelyrookewmde@mwmaint2002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https
* 14:37 ladsgroup@cumin1002: START - Cookbook sre.mysql.upgrade for pc1014.eqiad.wmnet
* 14:33 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
* 14:33 elukey@cumin1002: START - Cookbook sre.hosts.provision for host wikikube-worker2022.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 14:31 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 4%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71878 and previous config saved to /var/cache/conftool/dbconfig/20250108-143126-root.json
* 14:29 marostegui@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 75%: Repooling after cloning', diff saved to https://phabricator.wikimedia.org/P71877 and previous config saved to /var/cache/conftool/dbconfig/20250108-142923-root.json
* 14:23 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
* 14:21 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
* 14:16 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 3%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71876 and previous config saved to /var/cache/conftool/dbconfig/20250108-141620-root.json
* 14:14 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc2017.codfw.wmnet,pc[1014,1017].eqiad.wmnet with reason: Reboot
* 14:14 marostegui@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 50%: Repooling after cloning', diff saved to https://phabricator.wikimedia.org/P71875 and previous config saved to /var/cache/conftool/dbconfig/20250108-141418-root.json
* 14:14 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on pc2017.codfw.wmnet,pc[1014,1017].eqiad.wmnet with reason: Reboot
* 14:12 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
* 14:12 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
* 14:12 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2022.codfw.wmnet with OS bookworm
* 14:11 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2020.codfw.wmnet
* 14:11 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2020.codfw.wmnet
* 14:11 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2021.codfw.wmnet
* 14:11 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2021.codfw.wmnet
* 14:09 jelto: sudo homer 'cr*codfw*' commit 'T377877'
* 14:09 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
* 14:07 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 2%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71874 and previous config saved to /var/cache/conftool/dbconfig/20250108-140115-root.json
* 13:59 marostegui@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 25%: Repooling after cloning', diff saved to https://phabricator.wikimedia.org/P71873 and previous config saved to /var/cache/conftool/dbconfig/20250108-135913-root.json
* 13:46 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 1%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71872 and previous config saved to /var/cache/conftool/dbconfig/20250108-134610-root.json
* 13:45 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 100%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71871 and previous config saved to /var/cache/conftool/dbconfig/20250108-134544-root.json
* 13:44 marostegui@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 10%: Repooling after cloning', diff saved to https://phabricator.wikimedia.org/P71870 and previous config saved to /var/cache/conftool/dbconfig/20250108-134408-root.json
* 13:44 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2021.codfw.wmnet with OS bookworm
* 13:39 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2020.codfw.wmnet with OS bookworm
* 13:37 elukey: elukey@puppetserver1001:~$ sudo puppetserver ca clean --certname kubernetes1021.eqiad.wmnet
* 13:30 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 75%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71869 and previous config saved to /var/cache/conftool/dbconfig/20250108-133038-root.json
* 13:27 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling pc5 in codfw for test (T373037)', diff saved to https://phabricator.wikimedia.org/P71868 and previous config saved to /var/cache/conftool/dbconfig/20250108-132708-ladsgroup.json
* 13:25 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling pc5 in eqiad for test (T373037)', diff saved to https://phabricator.wikimedia.org/P71867 and previous config saved to /var/cache/conftool/dbconfig/20250108-132506-ladsgroup.json
* 13:23 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2021.codfw.wmnet with reason: host reimage
* 13:22 ladsgroup@deploy2002: Finished scap sync-world: Backport for gerrit:1108794Fully depool ParserCache section if load of the primary is zero (T373037 T383137) (duration: 19m 19s)
* 13:19 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 100%: Repooling after cloning', diff saved to https://phabricator.wikimedia.org/P71866 and previous config saved to /var/cache/conftool/dbconfig/20250108-131953-root.json
* 13:19 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2021.codfw.wmnet with reason: host reimage
* 13:17 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2020.codfw.wmnet with reason: host reimage
* 13:15 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 50%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71865 and previous config saved to /var/cache/conftool/dbconfig/20250108-131533-root.json
* 13:14 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2020.codfw.wmnet with reason: host reimage
* 13:13 ladsgroup@deploy2002: ladsgroup: Continuing with sync
* 13:12 ladsgroup@deploy2002: ladsgroup: Backport for gerrit:1108794Fully depool ParserCache section if load of the primary is zero (T373037 T383137) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 13:04 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 75%: Repooling after cloning', diff saved to https://phabricator.wikimedia.org/P71864 and previous config saved to /var/cache/conftool/dbconfig/20250108-130448-root.json
* 13:03 ladsgroup@deploy2002: Started scap sync-world: Backport for gerrit:1108794Fully depool ParserCache section if load of the primary is zero (T373037 T383137)
* 13:01 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2021
* 13:01 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2021
* 13:01 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2021
* 13:01 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2021.codfw.wmnet 210.32.192.10.in-addr.arpa 0.1.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:01 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2021.codfw.wmnet 210.32.192.10.in-addr.arpa 0.1.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 13:01 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:01 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2021 - jelto@cumin1002"
* 13:01 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2021 - jelto@cumin1002"
* 13:00 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 25%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71863 and previous config saved to /var/cache/conftool/dbconfig/20250108-130028-root.json
* 12:57 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 12:57 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2021
* 12:57 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2020
* 12:56 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2020
* 12:56 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2020
* 12:56 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2020.codfw.wmnet 208.32.192.10.in-addr.arpa 8.0.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 12:56 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2020.codfw.wmnet 208.32.192.10.in-addr.arpa 8.0.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 12:56 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:56 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2020 - jelto@cumin1002"
* 12:56 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2020 - jelto@cumin1002"
* 12:53 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1086.eqiad.wmnet
* 12:53 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1086.eqiad.wmnet
* 12:52 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 12:52 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2020
* 12:52 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2020.codfw.wmnet with OS bookworm
* 12:52 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2021.codfw.wmnet with OS bookworm
* 12:51 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2020-2021].codfw.wmnet
* 12:50 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2020-2021].codfw.wmnet
* 12:50 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
* 12:49 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 50%: Repooling after cloning', diff saved to https://phabricator.wikimedia.org/P71862 and previous config saved to /var/cache/conftool/dbconfig/20250108-124943-root.json
* 12:49 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2023.codfw.wmnet
* 12:49 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2023.codfw.wmnet
* 12:48 mvolz@deploy2002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 12:48 mvolz@deploy2002: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 12:47 mvolz@deploy2002: helmfile [codfw] DONE helmfile.d/services/citoid: apply
* 12:47 mvolz@deploy2002: helmfile [codfw] START helmfile.d/services/citoid: apply
* 12:46 root@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2128.codfw.wmnet onto db2228.codfw.wmnet
* 12:45 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2022.codfw.wmnet with OS bookworm
* 12:45 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 10%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71861 and previous config saved to /var/cache/conftool/dbconfig/20250108-124522-root.json
* 12:41 mvolz@deploy2002: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 12:40 mvolz@deploy2002: helmfile [staging] START helmfile.d/services/citoid: apply
* 12:39 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
* 12:34 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 25%: Repooling after cloning', diff saved to https://phabricator.wikimedia.org/P71860 and previous config saved to /var/cache/conftool/dbconfig/20250108-123437-root.json
* 12:30 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
* 12:30 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 5%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71859 and previous config saved to /var/cache/conftool/dbconfig/20250108-123017-root.json
* 12:21 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1086.eqiad.wmnet with OS bookworm
* 12:20 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
* 12:20 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
* 12:19 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 10%: Repooling after cloning', diff saved to https://phabricator.wikimedia.org/P71858 and previous config saved to /var/cache/conftool/dbconfig/20250108-121931-root.json
* 12:15 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 4%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71857 and previous config saved to /var/cache/conftool/dbconfig/20250108-121512-root.json
* 12:09 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1084-1085,1087].eqiad.wmnet
* 12:09 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1084-1085,1087].eqiad.wmnet
* 12:08 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
* 12:03 root@cumin1002: START - Cookbook sre.mysql.clone of db2128.codfw.wmnet onto db2228.codfw.wmnet
* 12:02 root@cumin1002: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2128.codfw.wmnet onto db2228.codfw.wmnet
* 12:02 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1086.eqiad.wmnet with reason: host reimage
* 12:00 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 3%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71856 and previous config saved to /var/cache/conftool/dbconfig/20250108-120006-root.json
* 11:59 root@cumin1002: START - Cookbook sre.mysql.clone of db2128.codfw.wmnet onto db2228.codfw.wmnet
* 11:59 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1086.eqiad.wmnet with reason: host reimage
* 11:59 marostegui@cumin1002: dbctl commit (dc=all): 'Add db2228 to dbctl depooled T373579', diff saved to https://phabricator.wikimedia.org/P71855 and previous config saved to /var/cache/conftool/dbconfig/20250108-115908-marostegui.json
* 11:53 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db[2128,2186].codfw.wmnet with reason: cloning
* 11:53 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db[2128,2186].codfw.wmnet with reason: cloning
* 11:52 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2128 T373579', diff saved to https://phabricator.wikimedia.org/P71854 and previous config saved to /var/cache/conftool/dbconfig/20250108-115206-marostegui.json
* 11:45 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 2%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71853 and previous config saved to /var/cache/conftool/dbconfig/20250108-114501-root.json
* 11:41 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1086
* 11:41 jayme@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1086
* 11:40 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1086.eqiad.wmnet with OS bookworm
* 11:40 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1087.eqiad.wmnet with OS bookworm
* 11:40 jayme@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker1086.eqiad.wmnet with OS bookworm
* 11:35 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1085.eqiad.wmnet with OS bookworm
* 11:34 moritzm: installing php7.4 security updates
* 11:31 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1084.eqiad.wmnet with OS bookworm
* 11:29 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 1%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71852 and previous config saved to /var/cache/conftool/dbconfig/20250108-112956-root.json
* 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on ganeti2027.codfw.wmnet with reason: reimage pending, blocked by T383207
* 11:28 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on ganeti2027.codfw.wmnet with reason: reimage pending, blocked by T383207
* 11:26 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ganeti2027.codfw.wmnet with reason: reimage pending, blocked by T383207
* 11:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 3:00:00 on ganeti2027.codfw.wmnet with reason: reimage pending, blocked by T383207
* 11:25 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
* 11:25 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
* 11:25 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2022.codfw.wmnet with OS bookworm
* 11:20 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1087.eqiad.wmnet with reason: host reimage
* 11:18 jelto: sudo homer 'lsw1-c6-codfw*' commit 'T377877'
* 11:17 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1085.eqiad.wmnet with reason: host reimage
* 11:13 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
* 11:13 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
* 11:12 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1084.eqiad.wmnet with reason: host reimage
* 11:10 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1085.eqiad.wmnet with reason: host reimage
* 11:09 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1087.eqiad.wmnet with reason: host reimage
* 11:08 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1084.eqiad.wmnet with reason: host reimage
* 11:08 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-airflow1002.eqiad.wmnet with reason: Migrating to kubernetes
* 11:07 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-airflow1002.eqiad.wmnet with reason: Migrating to kubernetes
* 11:06 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2022.codfw.wmnet with OS bookworm
* 11:06 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2023.codfw.wmnet with OS bookworm
* 11:00 moritzm: installing graphviz bugfix updates from bookworm point release
* 10:54 moritzm: installing numpy bugfix updates from bookworm point release
* 10:52 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1085
* 10:52 jayme@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1085
* 10:52 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1085.eqiad.wmnet with OS bookworm
* 10:52 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1087
* 10:52 jayme@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1087
* 10:52 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1087.eqiad.wmnet with OS bookworm
* 10:51 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1086
* 10:51 jayme@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1086
* 10:51 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1086.eqiad.wmnet with OS bookworm
* 10:51 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1084
* 10:51 jayme@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1084
* 10:51 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1084.eqiad.wmnet with OS bookworm
* 10:50 jayme@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1084.eqiad.wmnet wikikube-worker1085.eqiad.wmnet wikikube-worker1086.eqiad.wmnet wikikube-worker1087.eqiad.wmnet on all recursors
* 10:50 jayme@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1084.eqiad.wmnet wikikube-worker1085.eqiad.wmnet wikikube-worker1086.eqiad.wmnet wikikube-worker1087.eqiad.wmnet on all recursors
* 10:48 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1060 to wikikube-worker1085
* 10:47 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1085
* 10:47 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
* 10:47 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
* 10:47 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2022.codfw.wmnet with OS bookworm
* 10:46 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2022.codfw.wmnet with OS bookworm
* 10:46 moritzm: installing libnvme bugfix updates from bookworm point release
* 10:46 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2023.codfw.wmnet with reason: host reimage
* 10:44 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1085
* 10:44 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:43 moritzm: uploaded php7.4 1:7.4.33-1+0~20221108.73+debian10~1.gbpa00350a+wmf10u2+icu67u4 (backport of latest PHP security fixes to our PHP build) T378173
* 10:42 jayme@cumin1002: START - Cookbook sre.dns.netbox
* 10:42 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1059 to wikikube-worker1084
* 10:42 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1061 to wikikube-worker1086
* 10:42 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2023.codfw.wmnet with reason: host reimage
* 10:41 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1084
* 10:41 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1086
* 10:41 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1086
* 10:41 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:38 jayme@cumin1002: START - Cookbook sre.dns.netbox
* 10:38 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1084
* 10:38 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:37 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1062 to wikikube-worker1087
* 10:37 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
* 10:37 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
* 10:37 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2022.codfw.wmnet with OS bookworm
* 10:37 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1087
* 10:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ganeti2027.codfw.wmnet with reason: reimage to bookworm
* 10:37 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1087
* 10:37 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:37 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1062 to wikikube-worker1087 - jayme@cumin1002"
* 10:36 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 3:00:00 on ganeti2027.codfw.wmnet with reason: reimage to bookworm
* 10:36 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1062 to wikikube-worker1087 - jayme@cumin1002"
* 10:36 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2022.codfw.wmnet with OS bookworm
* 10:36 jayme@cumin1002: START - Cookbook sre.dns.netbox
* 10:34 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{wikikube-worker[1267-1269].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
* 10:34 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1269.eqiad.wmnet with OS bookworm
* 10:31 jayme@cumin1002: START - Cookbook sre.dns.netbox
* 10:31 jayme@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
* 10:30 jayme@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1062 to wikikube-worker1087
* 10:29 jayme@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1061 to wikikube-worker1086
* 10:29 jayme@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1060 to wikikube-worker1085
* 10:29 jayme@cumin1002: START - Cookbook sre.dns.netbox
* 10:28 jayme@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1059 to wikikube-worker1084
* 10:24 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
* 10:24 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
* 10:24 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 100%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71850 and previous config saved to /var/cache/conftool/dbconfig/20250108-102416-root.json
* 10:24 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2023
* 10:24 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2023
* 10:23 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2023
* 10:23 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2023.codfw.wmnet 213.32.192.10.in-addr.arpa 3.1.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:23 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2023.codfw.wmnet 213.32.192.10.in-addr.arpa 3.1.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 10:23 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:23 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2023 - jelto@cumin1002"
* 10:23 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2023 - jelto@cumin1002"
* 10:21 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2022.codfw.wmnet with OS bookworm
* 10:20 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 10:19 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2023
* 10:19 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2022.codfw.wmnet with OS bookworm
* 10:19 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.move-vlan (exit_code=99) for host wikikube-worker2022
* 10:19 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
* 10:19 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2023.codfw.wmnet with OS bookworm
* 10:19 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2022.codfw.wmnet with OS bookworm
* 10:15 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2022-2023].codfw.wmnet
* 10:15 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1269.eqiad.wmnet with reason: host reimage
* 10:14 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2022-2023].codfw.wmnet
* 10:12 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1269.eqiad.wmnet with reason: host reimage
* 10:09 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 75%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71849 and previous config saved to /var/cache/conftool/dbconfig/20250108-100910-root.json
* 10:04 moritzm: imported osmborder 0.1.0+wmf12u1 to apt.wikimedia.org/bookworm T381565
* 10:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet
* 10:02 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2024.codfw.wmnet
* 10:02 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2024.codfw.wmnet
* 10:02 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2025.codfw.wmnet
* 10:02 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2025.codfw.wmnet
* 10:00 jelto: sudo homer 'cr*codfw*' commit 'T377877'
* 10:00 jelto: sudo homer 'lsw1-c6-codfw*' commit 'T377877'
* 09:59 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2025.codfw.wmnet with OS bookworm
* 09:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2024.codfw.wmnet with OS bookworm
* 09:54 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 50%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71848 and previous config saved to /var/cache/conftool/dbconfig/20250108-095405-root.json
* 09:52 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1269.eqiad.wmnet with OS bookworm
* 09:50 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1268.eqiad.wmnet with OS bookworm
* 09:42 root@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2126.codfw.wmnet onto db2226.codfw.wmnet
* 09:39 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2025.codfw.wmnet with reason: host reimage
* 09:39 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 25%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71847 and previous config saved to /var/cache/conftool/dbconfig/20250108-093900-root.json
* 09:35 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2024.codfw.wmnet with reason: host reimage
* 09:31 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1268.eqiad.wmnet with reason: host reimage
* 09:30 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2024.codfw.wmnet with reason: host reimage
* 09:27 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2025.codfw.wmnet with reason: host reimage
* 09:25 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1268.eqiad.wmnet with reason: host reimage
* 09:23 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 10%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71846 and previous config saved to /var/cache/conftool/dbconfig/20250108-092354-root.json
* 09:13 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2024
* 09:13 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2024
* 09:13 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2024.codfw.wmnet with OS bookworm
* 09:12 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2024.codfw.wmnet with OS bookworm
* 09:10 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2025
* 09:10 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2025
* 09:10 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2025.codfw.wmnet with OS bookworm
* 09:08 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 5%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71845 and previous config saved to /var/cache/conftool/dbconfig/20250108-090849-root.json
* 09:08 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2025.codfw.wmnet with OS bookworm
* 09:05 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1268.eqiad.wmnet with OS bookworm
* 09:04 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1267.eqiad.wmnet with OS bookworm
* 08:53 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 4%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71844 and previous config saved to /var/cache/conftool/dbconfig/20250108-085344-root.json
* 08:44 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1267.eqiad.wmnet with reason: host reimage
* 08:42 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1267.eqiad.wmnet with reason: host reimage
* 08:41 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es2024.codfw.wmnet
* 08:41 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:41 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2024.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 08:41 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2024.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 08:38 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 3%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71843 and previous config saved to /var/cache/conftool/dbconfig/20250108-083838-root.json
* 08:38 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet
* 08:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet
* 08:37 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 08:36 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet
* 08:36 elukey: kill hanging processes on stat1011 to allow puppet to properly clean up absented users
* 08:32 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts es2024.codfw.wmnet
* 08:28 marostegui@cumin1002: dbctl commit (dc=all): 'Remove es2024 from dbctl T383028', diff saved to https://phabricator.wikimedia.org/P71842 and previous config saved to /var/cache/conftool/dbconfig/20250108-082807-marostegui.json
* 08:27 jmm@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Aitolkyn out of all services on: 2309 hosts
* 08:26 jmm@cumin2002: START - Cookbook sre.idm.logout Logging Aitolkyn out of all services on: 2309 hosts
* 08:23 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 2%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71841 and previous config saved to /var/cache/conftool/dbconfig/20250108-082333-root.json
* 08:23 elukey: destroy puppet cert for cloudelastic1011.eqiad.wmnet on puppetmaster1001 (cruft from old/wrong reimages)
* 08:21 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1267.eqiad.wmnet with OS bookworm
* 08:19 jayme@cumin1002: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P{wikikube-worker[1267-1269].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
* 08:08 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 1%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71840 and previous config saved to /var/cache/conftool/dbconfig/20250108-080828-root.json
* 07:51 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2024
* 07:51 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2024
* 07:51 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2024
* 07:51 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2024.codfw.wmnet 214.32.192.10.in-addr.arpa 4.1.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 07:51 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2024.codfw.wmnet 214.32.192.10.in-addr.arpa 4.1.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 07:51 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:51 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2024 - jelto@cumin1002"
* 07:51 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2024 - jelto@cumin1002"
* 07:50 moritzm: truncate /var/log/debug on seaborgium to unblock some disk space
* 07:49 root@cumin1002: START - Cookbook sre.mysql.clone of db2126.codfw.wmnet onto db2226.codfw.wmnet
* 07:48 marostegui@cumin1002: dbctl commit (dc=all): 'Add db2226 depooled', diff saved to https://phabricator.wikimedia.org/P71839 and previous config saved to /var/cache/conftool/dbconfig/20250108-074856-marostegui.json
* 07:47 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 07:47 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2024
* 07:47 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2025
* 07:47 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2025
* 07:47 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2025.codfw.wmnet with OS bookworm
* 07:47 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2024.codfw.wmnet with OS bookworm
* 07:45 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2024-2025].codfw.wmnet
* 07:44 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2024-2025].codfw.wmnet
* 07:41 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: cloning
* 07:41 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: cloning
* 07:38 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2126.codfw.wmnet with reason: cloning
* 07:38 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2126.codfw.wmnet with reason: cloning
* 07:36 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2126 T373579', diff saved to https://phabricator.wikimedia.org/P71838 and previous config saved to /var/cache/conftool/dbconfig/20250108-073603-marostegui.json
* 07:27 marostegui@cumin1002: dbctl commit (dc=all): 'es1020 (re)pooling @ 100%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71837 and previous config saved to /var/cache/conftool/dbconfig/20250108-072733-root.json
* 07:12 marostegui@cumin1002: dbctl commit (dc=all): 'es1020 (re)pooling @ 75%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71836 and previous config saved to /var/cache/conftool/dbconfig/20250108-071228-root.json
* 06:57 marostegui@cumin1002: dbctl commit (dc=all): 'es1020 (re)pooling @ 50%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71834 and previous config saved to /var/cache/conftool/dbconfig/20250108-065723-root.json
* 06:53 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dbproxy1021.eqiad.wmnet
* 06:53 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 06:53 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy1021.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 06:53 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy1021.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 06:50 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 06:44 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts dbproxy1021.eqiad.wmnet
* 06:42 marostegui@cumin1002: dbctl commit (dc=all): 'es1020 (re)pooling @ 25%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71833 and previous config saved to /var/cache/conftool/dbconfig/20250108-064217-root.json
* 06:27 marostegui@cumin1002: dbctl commit (dc=all): 'es1020 (re)pooling @ 10%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71832 and previous config saved to /var/cache/conftool/dbconfig/20250108-062712-root.json
* 06:25 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es1021.eqiad.wmnet with reason: cloning es1042
* 06:25 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on es1021.eqiad.wmnet with reason: cloning es1042
* 06:24 marostegui@cumin1002: dbctl commit (dc=all): 'Add es1041 to dbctl depooled T382569', diff saved to https://phabricator.wikimedia.org/P71831 and previous config saved to /var/cache/conftool/dbconfig/20250108-062447-marostegui.json
* 06:19 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1021 T382569', diff saved to https://phabricator.wikimedia.org/P71830 and previous config saved to /var/cache/conftool/dbconfig/20250108-061928-marostegui.json
* 06:19 marostegui@cumin1002: dbctl commit (dc=all): 'Switchover es4 eqiad master dbmaint T382569', diff saved to https://phabricator.wikimedia.org/P71829 and previous config saved to /var/cache/conftool/dbconfig/20250108-061914-marostegui.json
* 06:12 marostegui@cumin1002: dbctl commit (dc=all): 'es1020 (re)pooling @ 5%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71828 and previous config saved to /var/cache/conftool/dbconfig/20250108-061207-root.json
== 2025-01-07 ==
* 23:58 eileen: civicrm upgraded from to 
* 23:49 urbanecm@deploy2002: Finished scap sync-world: Backport for [[gerrit:1108724[Growth] enwiki: Deploy Add Link to 5% of users (T382382)]] (duration: 13m 34s)
* 23:42 urbanecm@deploy2002: urbanecm: Continuing with sync
* 23:42 urbanecm@deploy2002: urbanecm: Backport for [[gerrit:1108724[Growth] enwiki: Deploy Add Link to 5% of users (T382382)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 23:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
* 23:36 urbanecm@deploy2002: Started scap sync-world: Backport for [[gerrit:1108724[Growth] enwiki: Deploy Add Link to 5% of users (T382382)]]
* 23:35 cdanis@deploy2002: Finished scap sync-world: Backport for gerrit:1108850tracing: Disable tracing in CLI mode (T340552) (duration: 14m 23s)
* 23:27 cdanis@deploy2002: cdanis: Continuing with sync
* 23:27 cdanis@deploy2002: cdanis: Backport for gerrit:1108850tracing: Disable tracing in CLI mode (T340552) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 23:20 cdanis@deploy2002: Started scap sync-world: Backport for gerrit:1108850tracing: Disable tracing in CLI mode (T340552)
* 23:19 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 23:16 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
* 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:47 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:35 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:22 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:22 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:15 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 22:09 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
* 22:02 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
* 20:56 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
* 20:56 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
* 20:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
* 20:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
* 20:46 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on doc1003.eqiad.wmnet with reason: maintenance
* 20:45 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on doc1003.eqiad.wmnet with reason: maintenance
* 20:19 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
* 20:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
* 20:16 cdanis@deploy2002: Finished scap sync-world: Backport for gerrit:1108802group0: enable OpenTelemetry exports (T340552) (duration: 16m 06s)
* 20:15 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'.
* 20:15 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'.
* 20:08 cdanis@deploy2002: cdanis: Continuing with sync
* 20:07 cdanis@deploy2002: cdanis: Backport for gerrit:1108802group0: enable OpenTelemetry exports (T340552) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 20:00 cdanis@deploy2002: Started scap sync-world: Backport for gerrit:1108802group0: enable OpenTelemetry exports (T340552)
* 20:00 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
* 20:00 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
* 19:56 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
* 19:55 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
* 19:52 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply
* 19:52 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply
* 19:36 dduvall@deploy2002: rebuilt and synchronized wikiversions files: group0 to 1.44.0-wmf.11 refs T382362
* 18:41 xcollazo@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
* 18:41 xcollazo@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
* 18:27 xcollazo@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: sync
* 18:27 xcollazo@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: sync
* 18:17 xcollazo@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
* 18:17 xcollazo@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
* 18:12 mvernon@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8 days, 0:00:00 on ms-be2075.codfw.wmnet with reason: host is awaiting attention from Dell
* 18:11 mvernon@cumin1002: START - Cookbook sre.hosts.downtime for 8 days, 0:00:00 on ms-be2075.codfw.wmnet with reason: host is awaiting attention from Dell
* 17:50 ottomata: Disable varnish handling of /beacon/event to decommission eventlogging backend [puppet] - T238230 T353817
* 17:49 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply
* 17:49 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply
* 17:43 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
* 17:43 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
* 17:29 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply
* 17:29 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply
* 17:26 dzahn@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Muhammad Jaziraly out of all services on: 4 hosts
* 17:26 dzahn@cumin2002: START - Cookbook sre.idm.logout Logging Muhammad Jaziraly out of all services on: 4 hosts
* 17:24 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
* 17:24 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
* 17:20 dzahn@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Muhammad Jaziraly out of all services on: 2313 hosts
* 17:18 dzahn@cumin2002: START - Cookbook sre.idm.logout Logging Muhammad Jaziraly out of all services on: 2313 hosts
* 16:53 ejegg: fundraising civicrm upgraded from to 
* 16:44 jayme: puppet ca destroy citoid.discovery.wmnet - T381474
* 16:44 jayme: puppet ca destroy mathoid.discovery.wmnet - T381474
* 16:44 jayme: puppet ca destroy termbox.discovery.wmnet - T381474
* 16:43 mutante: krb1001 - sudo manage_principals.py delete muja@WIKIMEDIA (T383056)
* 16:32 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
* 16:32 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-video: apply
* 16:31 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply
* 16:31 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply
* 16:31 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
* 16:30 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply
* 16:30 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply
* 16:30 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-media: apply
* 16:30 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply
* 16:30 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply
* 16:30 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox: apply
* 16:29 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox: apply
* 16:29 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
* 16:28 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
* 16:28 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
* 16:27 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
* 16:27 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
* 16:27 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply
* 16:27 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
* 16:27 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
* 16:26 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
* 16:26 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
* 16:26 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
* 16:26 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox: apply
* 16:25 lucaswerkmeister-wmde@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
* 16:24 lucaswerkmeister-wmde@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-video: apply
* 16:24 lucaswerkmeister-wmde@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
* 16:24 lucaswerkmeister-wmde@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
* 16:24 lucaswerkmeister-wmde@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
* 16:23 lucaswerkmeister-wmde@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply
* 16:23 lucaswerkmeister-wmde@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
* 16:23 lucaswerkmeister-wmde@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-media: apply
* 16:22 lucaswerkmeister-wmde@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
* 16:22 lucaswerkmeister-wmde@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
* 16:22 lucaswerkmeister-wmde@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox: apply
* 16:22 lucaswerkmeister-wmde@deploy2002: helmfile [staging] START helmfile.d/services/shellbox: apply
* 16:06 elukey: reloaded postgres config on puppetdb1003 to pick up new wal size settings
* 15:59 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2026.codfw.wmnet
* 15:59 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2026.codfw.wmnet
* 15:58 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2027.codfw.wmnet
* 15:58 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2027.codfw.wmnet
* 15:58 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2026.codfw.wmnet with OS bookworm
* 15:58 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply
* 15:57 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply
* 15:57 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
* 15:57 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
* 15:55 lucaswerkmeister-wmde@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
* 15:55 lucaswerkmeister-wmde@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
* 15:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2027.codfw.wmnet with OS bookworm
* 15:51 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{wikikube-worker[1264-1266].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
* 15:51 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1266.eqiad.wmnet with OS bookworm
* 15:38 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2026.codfw.wmnet with reason: host reimage
* 15:35 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2027.codfw.wmnet with reason: host reimage
* 15:31 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1266.eqiad.wmnet with reason: host reimage
* 15:31 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{wikikube-worker[1253-1255].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
* 15:31 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1255.eqiad.wmnet with OS bookworm
* 15:29 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2027.codfw.wmnet with reason: host reimage
* 15:29 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2026.codfw.wmnet with reason: host reimage
* 15:26 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1266.eqiad.wmnet with reason: host reimage
* 15:24 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: sync
* 15:24 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: sync
* 15:13 Lucas_WMDE: UTC afternoon backport+config window done
* 15:13 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1108459Enable AutoModerator on zhwiki (T367306), gerrit:1108525cswikivoyage: Change the wordmark v2 (T382779) (duration: 30m 38s)
* 15:11 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2026
* 15:11 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2026
* 15:11 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2027
* 15:11 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2027
* 15:11 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1255.eqiad.wmnet with reason: host reimage
* 15:11 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2026.codfw.wmnet with OS bookworm
* 15:11 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2027.codfw.wmnet with OS bookworm
* 15:09 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2026-2027].codfw.wmnet
* 15:08 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2026-2027].codfw.wmnet
* 15:08 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1255.eqiad.wmnet with reason: host reimage
* 15:07 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2028.codfw.wmnet
* 15:07 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2028.codfw.wmnet
* 15:07 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2029.codfw.wmnet
* 15:07 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2029.codfw.wmnet
* 15:05 lucaswerkmeister-wmde@deploy2002: zhaofjx, lucaswerkmeister-wmde: Continuing with sync
* 14:58 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2029.codfw.wmnet with OS bookworm
* 14:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2028.codfw.wmnet with OS bookworm
* 14:51 moritzm: installing intel-microcode security updates
* 14:49 lucaswerkmeister-wmde@deploy2002: zhaofjx, lucaswerkmeister-wmde: Backport for gerrit:1108459Enable AutoModerator on zhwiki (T367306), gerrit:1108525cswikivoyage: Change the wordmark v2 (T382779) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:42 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1108459Enable AutoModerator on zhwiki (T367306), gerrit:1108525cswikivoyage: Change the wordmark v2 (T382779)
* 14:37 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2029.codfw.wmnet with reason: host reimage
* 14:36 ladsgroup@deploy2002: Finished scap sync-world: Backport for gerrit:1100850Reactivate Parsoid+Kartographer on hewiki and commonswiki (T373454 T373460) (duration: 24m 56s)
* 14:34 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2028.codfw.wmnet with reason: host reimage
* 14:34 elukey@deploy2002: helmfile [staging] DONE helmfile.d/services/kartotherian: sync
* 14:33 elukey@deploy2002: helmfile [staging] START helmfile.d/services/kartotherian: sync
* 14:31 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2029.codfw.wmnet with reason: host reimage
* 14:31 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2028.codfw.wmnet with reason: host reimage
* 14:26 ladsgroup@deploy2002: ladsgroup, ihurbain: Continuing with sync
* 14:20 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1266.eqiad.wmnet with OS bookworm
* 14:20 ladsgroup@deploy2002: ladsgroup, ihurbain: Backport for gerrit:1100850Reactivate Parsoid+Kartographer on hewiki and commonswiki (T373454 T373460) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:18 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1265.eqiad.wmnet with OS bookworm
* 14:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1001.eqiad.wmnet
* 14:16 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1255.eqiad.wmnet with OS bookworm
* 14:14 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1254.eqiad.wmnet with OS bookworm
* 14:13 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2029
* 14:13 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2029
* 14:13 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2028
* 14:13 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2028
* 14:13 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2029.codfw.wmnet with OS bookworm
* 14:13 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2028.codfw.wmnet with OS bookworm
* 14:12 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2028-2029].codfw.wmnet
* 14:11 ladsgroup@deploy2002: Started scap sync-world: Backport for gerrit:1100850Reactivate Parsoid+Kartographer on hewiki and commonswiki (T373454 T373460)
* 14:11 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host sretest1001.eqiad.wmnet
* 14:10 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2028-2029].codfw.wmnet
* 14:08 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2031.codfw.wmnet
* 14:08 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2031.codfw.wmnet
* 14:08 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2030.codfw.wmnet
* 14:08 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2030.codfw.wmnet
* 14:02 elukey: re-enable puppet fleetwide after maintenance
* 13:59 jayme@cumin1002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on wikikube-worker1265.eqiad.wmnet with reason: host reimage
* 13:59 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1265.eqiad.wmnet with reason: host reimage
* 13:57 elukey: start postgres and puppetdb on puppetdb2003
* 13:54 jayme@cumin1002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on wikikube-worker1254.eqiad.wmnet with reason: host reimage
* 13:53 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1254.eqiad.wmnet with reason: host reimage
* 13:52 elukey@cumin1002: END (FAIL) - Cookbook sre.postgresql.postgres-init (exit_code=99)
* 13:41 elukey@cumin1002: START - Cookbook sre.postgresql.postgres-init
* 13:39 elukey: stop puppetdb and postgres on puppetdb2003 - T383114
* 13:39 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1265.eqiad.wmnet with OS bookworm
* 13:35 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on puppetdb2003.codfw.wmnet with reason: Resync postgres
* 13:35 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on puppetdb2003.codfw.wmnet with reason: Resync postgres
* 13:35 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1264.eqiad.wmnet with OS bookworm
* 13:33 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1254.eqiad.wmnet with OS bookworm
* 13:32 elukey: disable puppet fleetwide to allow maintenance on puppetdb2003
* 13:31 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1253.eqiad.wmnet with OS bookworm
* 13:30 moritzm: install gst-plugins-base1.0 security updates
* 13:30 Amir1: running mwscript purgeParserCache.php --wiki=aawiki --tag pc5 --age=2592000 --msleep 200 in *eqiad* (T382948)
* 13:15 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1264.eqiad.wmnet with reason: host reimage
* 13:12 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1253.eqiad.wmnet with reason: host reimage
* 13:09 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1264.eqiad.wmnet with reason: host reimage
* 13:08 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1253.eqiad.wmnet with reason: host reimage
* 12:49 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1264.eqiad.wmnet with OS bookworm
* 12:48 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1253.eqiad.wmnet with OS bookworm
* 12:47 jayme@cumin1002: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P{wikikube-worker[1264-1266].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
* 12:47 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2030.codfw.wmnet with OS bookworm
* 12:46 jayme@cumin1002: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P{wikikube-worker[1253-1255].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
* 12:44 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2031.codfw.wmnet with OS bookworm
* 12:26 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2030.codfw.wmnet with reason: host reimage
* 12:23 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2031.codfw.wmnet with reason: host reimage
* 12:20 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2030.codfw.wmnet with reason: host reimage
* 12:19 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2031.codfw.wmnet with reason: host reimage
* 12:03 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dbproxy1020.eqiad.wmnet
* 12:03 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:03 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy1020.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 12:02 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy1020.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 12:02 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2030
* 12:02 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2030
* 12:02 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2031
* 12:02 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2031
* 12:01 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2030.codfw.wmnet with OS bookworm
* 12:01 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2031.codfw.wmnet with OS bookworm
* 12:00 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2030-2031].codfw.wmnet
* 11:59 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2030-2031].codfw.wmnet
* 11:58 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 11:58 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2032.codfw.wmnet
* 11:58 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2032.codfw.wmnet
* 11:57 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2033.codfw.wmnet
* 11:57 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2033.codfw.wmnet
* 11:56 jelto: sudo homer 'cr*codfw*' commit 'T377877'
* 11:55 jelto: sudo homer 'lsw1-c6-codfw*' commit 'T377877'
* 11:52 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts dbproxy1020.eqiad.wmnet
* 11:52 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2032.codfw.wmnet with OS bookworm
* 11:46 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'.
* 11:46 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'.
* 11:41 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2033.codfw.wmnet with OS bookworm
* 11:32 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2032.codfw.wmnet with reason: host reimage
* 11:26 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2032.codfw.wmnet with reason: host reimage
* 11:25 moritzm: refreshed Ganeti internal cert for ulsfo (after adding a manual temp cert to unblock itself) T382873
* 11:22 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2033.codfw.wmnet with reason: host reimage
* 11:18 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2033.codfw.wmnet with reason: host reimage
* 11:13 hashar@deploy2002: Finished deploy [integration/docroot@5d32766]: Remove Flow from being listed on doc.wikimedia.org frontpage - T379671 (duration: 00m 11s)
* 11:13 hashar@deploy2002: Started deploy [integration/docroot@5d32766]: Remove Flow from being listed on doc.wikimedia.org frontpage - T379671
* 11:08 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2032
* 11:08 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2032
* 11:08 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2032
* 11:08 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2032.codfw.wmnet 215.32.192.10.in-addr.arpa 5.1.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 11:08 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2032.codfw.wmnet 215.32.192.10.in-addr.arpa 5.1.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 11:08 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:07 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2032 - jelto@cumin1002"
* 11:07 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2032 - jelto@cumin1002"
* 11:03 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 11:02 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2032
* 11:01 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2032.codfw.wmnet with OS bookworm
* 11:01 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2033
* 11:01 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2033
* 11:01 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2033.codfw.wmnet with OS bookworm
* 10:56 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2032-2033].codfw.wmnet
* 10:53 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2032-2033].codfw.wmnet
* 10:52 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2034.codfw.wmnet
* 10:52 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2034.codfw.wmnet
* 10:52 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2035.codfw.wmnet
* 10:51 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2035.codfw.wmnet
* 10:51 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ganeti[4005-4008].ulsfo.wmnet with reason: renew certs
* 10:50 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2035.codfw.wmnet with OS bookworm
* 10:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 3:00:00 on ganeti[4005-4008].ulsfo.wmnet with reason: renew certs
* 10:50 cmooney@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 269180
* 10:49 cmooney@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 269180
* 10:46 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2034.codfw.wmnet with OS bookworm
* 10:30 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2035.codfw.wmnet with reason: host reimage
* 10:26 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2034.codfw.wmnet with reason: host reimage
* 10:25 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2035.codfw.wmnet with reason: host reimage
* 10:23 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2034.codfw.wmnet with reason: host reimage
* 10:05 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2035
* 10:05 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2035
* 10:05 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2034
* 10:05 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2034
* 10:05 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2034.codfw.wmnet with OS bookworm
* 10:05 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2035.codfw.wmnet with OS bookworm
* 10:05 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2003.codfw.wmnet
* 10:03 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2034-2035].codfw.wmnet
* 10:02 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2034-2035].codfw.wmnet
* 10:01 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2036.codfw.wmnet
* 10:01 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2036.codfw.wmnet
* 10:01 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2037.codfw.wmnet
* 10:01 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2037.codfw.wmnet
* 10:01 cmooney@cumin1002: START - Cookbook sre.hosts.reboot-single for host rpki2003.codfw.wmnet
* 09:51 jelto: sudo homer 'cr*codfw*' commit 'T377877'
* 09:50 jelto: sudo homer 'lsw1-c1-codfw*' commit 'T377877'
* 09:46 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2037.codfw.wmnet with OS bookworm
* 09:40 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2036.codfw.wmnet with OS bookworm
* 09:26 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2037.codfw.wmnet with reason: host reimage
* 09:22 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2037.codfw.wmnet with reason: host reimage
* 09:20 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2036.codfw.wmnet with reason: host reimage
* 09:17 moritzm: installing systemd bugfix updates from Bookworm point release
* 09:16 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2036.codfw.wmnet with reason: host reimage
* 09:08 moritzm: installing Java 21.0.5 security updates
* 09:02 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2037
* 09:02 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2037
* 09:02 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2037
* 09:02 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2037.codfw.wmnet 146.32.192.10.in-addr.arpa 6.4.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 09:02 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2037.codfw.wmnet 146.32.192.10.in-addr.arpa 6.4.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 09:02 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:02 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2037 - jelto@cumin1002"
* 09:02 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2037 - jelto@cumin1002"
* 08:59 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 08:58 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2037
* 08:57 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2037.codfw.wmnet with OS bookworm
* 08:57 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2036
* 08:57 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2036
* 08:56 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2036.codfw.wmnet with OS bookworm
* 08:54 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2036-2037].codfw.wmnet
* 08:53 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2036-2037].codfw.wmnet
* 08:48 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2038.codfw.wmnet
* 08:48 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2038.codfw.wmnet
* 08:48 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2039.codfw.wmnet
* 08:48 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet
* 08:47 jelto: sudo homer 'lsw1-c5-codfw*' commit 'T377877'
* 08:44 jelto: sudo homer 'cr*codfw*' commit 'T377877'
* 08:44 jelto: sudo homer 'lsw1-c1-codfw*' commit 'T377877'
* 08:43 jelto: sudo homer 'lsw1-c7-codfw*' commit 'T377877'
* 08:41 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es1020.eqiad.wmnet with reason: cloning es1041
* 08:41 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on es1020.eqiad.wmnet with reason: cloning es1041
* 08:39 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1020 T382569', diff saved to https://phabricator.wikimedia.org/P71822 and previous config saved to /var/cache/conftool/dbconfig/20250107-083930-marostegui.json
* 08:38 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2038.codfw.wmnet with OS bookworm
* 08:38 marostegui@cumin1002: dbctl commit (dc=all): 'Switchover es5 eqiad master dbmaint T382569', diff saved to https://phabricator.wikimedia.org/P71821 and previous config saved to /var/cache/conftool/dbconfig/20250107-083811-marostegui.json
* 08:26 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2039.codfw.wmnet with OS bookworm
* 08:20 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es2025.codfw.wmnet
* 08:20 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:20 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2025.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 08:19 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2025.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 08:17 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2038.codfw.wmnet with reason: host reimage
* 08:14 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 08:13 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2038.codfw.wmnet with reason: host reimage
* 08:09 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts es2025.codfw.wmnet
* 08:06 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2039.codfw.wmnet with reason: host reimage
* 08:03 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2039.codfw.wmnet with reason: host reimage
* 07:53 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2038
* 07:53 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2038
* 07:53 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2038
* 07:53 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2038.codfw.wmnet 147.32.192.10.in-addr.arpa 7.4.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 07:53 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2038.codfw.wmnet 147.32.192.10.in-addr.arpa 7.4.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 07:53 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:53 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2038 - jelto@cumin1002"
* 07:53 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2038 - jelto@cumin1002"
* 07:49 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 07:49 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2038
* 07:49 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2038.codfw.wmnet with OS bookworm
* 07:43 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2039
* 07:43 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2039
* 07:43 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2039
* 07:43 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2039.codfw.wmnet 150.32.192.10.in-addr.arpa 0.5.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 07:43 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2039.codfw.wmnet 150.32.192.10.in-addr.arpa 0.5.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 07:43 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 07:42 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2039 - jelto@cumin1002"
* 07:41 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2039 - jelto@cumin1002"
* 07:37 jelto@cumin1002: START - Cookbook sre.dns.netbox
* 07:37 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2039
* 07:37 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2039.codfw.wmnet with OS bookworm
* 07:33 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2038-2039].codfw.wmnet
* 07:32 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2038-2039].codfw.wmnet
* 06:50 marostegui@cumin1002: dbctl commit (dc=all): 'Remove es2025 from dbctl T381848', diff saved to https://phabricator.wikimedia.org/P71820 and previous config saved to /var/cache/conftool/dbconfig/20250107-064958-marostegui.json
* 05:40 marostegui: Switchover m5 eqiad proxy dbmaint eqiad T368874
* 04:51 mwpresync@deploy2002: Finished scap sync-world: testwikis to 1.44.0-wmf.11 refs T382362 (duration: 48m 35s)
* 04:02 mwpresync@deploy2002: Started scap sync-world: testwikis to 1.44.0-wmf.11 refs T382362
* 00:24 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{wikikube-worker[1257-1263].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
* 00:24 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1263.eqiad.wmnet with OS bookworm
* 00:05 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1263.eqiad.wmnet with reason: host reimage
* 00:01 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1263.eqiad.wmnet with reason: host reimage
== 2025-01-06 ==
* 23:41 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1263.eqiad.wmnet with OS bookworm
* 23:39 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1262.eqiad.wmnet with OS bookworm
* 23:19 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1262.eqiad.wmnet with reason: host reimage
* 23:16 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1262.eqiad.wmnet with reason: host reimage
* 22:56 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1262.eqiad.wmnet with OS bookworm
* 22:55 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS bookworm
* 22:36 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage
* 22:33 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage
* 22:13 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS bookworm
* 22:10 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1260.eqiad.wmnet with OS bookworm
* 21:49 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1260.eqiad.wmnet with reason: host reimage
* 21:45 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1260.eqiad.wmnet with reason: host reimage
* 21:39 cjming: end of UTC late backport window
* 21:31 cjming@deploy2002: Finished scap sync-world: Backport for gerrit:1108130Move logic for type infering to server (T382042) (duration: 19m 31s)
* 21:25 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1260.eqiad.wmnet with OS bookworm
* 21:25 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1259.eqiad.wmnet with OS bookworm
* 21:22 cjming@deploy2002: cjming, jdlrobson: Continuing with sync
* 21:16 cjming@deploy2002: cjming, jdlrobson: Backport for gerrit:1108130Move logic for type infering to server (T382042) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:12 cjming@deploy2002: Started scap sync-world: Backport for gerrit:1108130Move logic for type infering to server (T382042)
* 21:06 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1259.eqiad.wmnet with reason: host reimage
* 21:02 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1259.eqiad.wmnet with reason: host reimage
* 20:49 eileen: tools upgraded from to 
* 20:41 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1259.eqiad.wmnet with OS bookworm
* 20:39 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1258.eqiad.wmnet with OS bookworm
* 20:24 brett: running authdns-update for CR 
* 20:19 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1258.eqiad.wmnet with reason: host reimage
* 20:15 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1258.eqiad.wmnet with reason: host reimage
* 19:55 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1258.eqiad.wmnet with OS bookworm
* 19:55 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1257.eqiad.wmnet with OS bookworm
* 19:36 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1257.eqiad.wmnet with reason: host reimage
* 19:32 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1257.eqiad.wmnet with reason: host reimage
* 19:11 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1257.eqiad.wmnet with OS bookworm
* 19:09 kamila@cumin1002: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P{wikikube-worker[1257-1263].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
* 18:47 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1244.eqiad.wmnet
* 18:47 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1244.eqiad.wmnet
* 18:32 ChrisDobbins901_: cdobbins@dns1004 running authdns-update for CR 
* 18:31 ChrisDobbins901_: cdobbins@cumin1002 running authdns-update for CR 
* 17:59 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1244.eqiad.wmnet with OS bookworm
* 17:40 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T370903)', diff saved to https://phabricator.wikimedia.org/P71818 and previous config saved to /var/cache/conftool/dbconfig/20250106-174024-ladsgroup.json
* 17:39 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1244.eqiad.wmnet with reason: host reimage
* 17:37 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1244.eqiad.wmnet with reason: host reimage
* 17:25 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P71817 and previous config saved to /var/cache/conftool/dbconfig/20250106-172517-ladsgroup.json
* 17:16 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1244.eqiad.wmnet with OS bookworm
* 17:15 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{wikikube-worker[1250-1252].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
* 17:15 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1252.eqiad.wmnet with OS bookworm
* 17:15 jayme@cumin1002: END (FAIL) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=1) rolling reimage on P{wikikube-worker[1240-1244].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
* 17:15 jayme@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker1244.eqiad.wmnet with OS bookworm
* 17:10 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P71816 and previous config saved to /var/cache/conftool/dbconfig/20250106-171010-ladsgroup.json
* 17:00 jdrewniak@deploy2002: Synchronized portals: Wikimedia Portals Update: gerrit:1108451 Bumping portals to master (T128546) (duration: 02m 43s)
* 16:58 jdrewniak@deploy2002: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: gerrit:1108451 Bumping portals to master (T128546) (duration: 12m 29s)
* 16:56 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1252.eqiad.wmnet with reason: host reimage
* 16:55 dani@deploy2002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 16:55 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T370903)', diff saved to https://phabricator.wikimedia.org/P71815 and previous config saved to /var/cache/conftool/dbconfig/20250106-165503-ladsgroup.json
* 16:54 dani@deploy2002: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 16:54 dani@deploy2002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 16:54 dani@deploy2002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 16:54 dani@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 16:53 dani@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
* 16:52 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1252.eqiad.wmnet with reason: host reimage
* 16:42 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1193 (T370903)', diff saved to https://phabricator.wikimedia.org/P71813 and previous config saved to /var/cache/conftool/dbconfig/20250106-164215-ladsgroup.json
* 16:42 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1193.eqiad.wmnet with reason: Maintenance
* 16:42 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1193.eqiad.wmnet with reason: Maintenance
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 16:38 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 16:37 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 15:51 moritzm: uploaded openjdk-21 21.0.5+11-1~deb12u1 to apt.wikimedia.org component/jdk21
* 15:39 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2143.codfw.wmnet with reason: onsite maintenance
* 15:38 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2143.codfw.wmnet with reason: onsite maintenance
* 15:31 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2041.codfw.wmnet
* 15:31 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2041.codfw.wmnet
* 15:31 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2040.codfw.wmnet
* 15:31 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2040.codfw.wmnet
* 15:29 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2040.codfw.wmnet with OS bookworm
* 15:27 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1252.eqiad.wmnet with OS bookworm
* 15:25 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1251.eqiad.wmnet with OS bookworm
* 15:23 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1244.eqiad.wmnet with OS bookworm
* 15:23 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2041.codfw.wmnet with OS bookworm
* 15:13 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 15:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 15:12 Lucas_WMDE: UTC afternoon backport+config window done
* 15:11 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1100083Remove EntitySchema DataType feature flag - is always enabled (T333667) (duration: 13m 25s)
* 15:09 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2040.codfw.wmnet with reason: host reimage
* 15:06 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1251.eqiad.wmnet with reason: host reimage
* 15:04 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, arthurtaylor: Continuing with sync
* 15:03 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2041.codfw.wmnet with reason: host reimage
* 15:02 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, arthurtaylor: Backport for gerrit:1100083Remove EntitySchema DataType feature flag - is always enabled (T333667) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 15:00 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1251.eqiad.wmnet with reason: host reimage
* 15:00 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T371742)', diff saved to https://phabricator.wikimedia.org/P71811 and previous config saved to /var/cache/conftool/dbconfig/20250106-150040-ladsgroup.json
* 14:58 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2040.codfw.wmnet with reason: host reimage
* 14:58 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2041.codfw.wmnet with reason: host reimage
* 14:58 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1100083Remove EntitySchema DataType feature flag - is always enabled (T333667)
* 14:56 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1108403Add bfw, gju-arab, gju-deva, hoc and kgg to wmgExtraLanguageNames (T381934) (duration: 17m 11s)
* 14:49 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, jhsoby: Continuing with sync
* 14:46 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, jhsoby: Backport for gerrit:1108403Add bfw, gju-arab, gju-deva, hoc and kgg to wmgExtraLanguageNames (T381934) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:45 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P71810 and previous config saved to /var/cache/conftool/dbconfig/20250106-144534-ladsgroup.json
* 14:41 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2040
* 14:40 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2040
* 14:40 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2041
* 14:40 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2041
* 14:40 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2040.codfw.wmnet with OS bookworm
* 14:40 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2041.codfw.wmnet with OS bookworm
* 14:40 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1251.eqiad.wmnet with OS bookworm
* 14:39 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1108403Add bfw, gju-arab, gju-deva, hoc and kgg to wmgExtraLanguageNames (T381934)
* 14:39 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2040-2041].codfw.wmnet
* 14:38 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1250.eqiad.wmnet with OS bookworm
* 14:38 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1106038ExtensionDistributor: Mark 1.43 as stable; remove 1.41 as EOL (T372331 T376550) (duration: 14m 14s)
* 14:35 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2040-2041].codfw.wmnet
* 14:34 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2042.codfw.wmnet
* 14:34 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2042.codfw.wmnet
* 14:34 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2043.codfw.wmnet
* 14:34 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2043.codfw.wmnet
* 14:33 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2042.codfw.wmnet with OS bookworm
* 14:33 jayme@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1243.eqiad.wmnet with OS bookworm
* 14:30 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2043.codfw.wmnet with OS bookworm
* 14:30 lucaswerkmeister-wmde@deploy2002: macfan4000, lucaswerkmeister-wmde: Continuing with sync
* 14:30 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P71809 and previous config saved to /var/cache/conftool/dbconfig/20250106-143027-ladsgroup.json
* 14:28 moritzm: installing libvirt bugfix updates
* 14:27 lucaswerkmeister-wmde@deploy2002: macfan4000, lucaswerkmeister-wmde: Backport for gerrit:1106038ExtensionDistributor: Mark 1.43 as stable; remove 1.41 as EOL (T372331 T376550) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:23 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1106038ExtensionDistributor: Mark 1.43 as stable; remove 1.41 as EOL (T372331 T376550)
* 14:22 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for gerrit:1107028Add mergehistory to import and transwiki on en.wikibooks (T382785), gerrit:1108171Add suppressredirect and delete-redirect to en.wikinews reviewers (T382887) (duration: 15m 02s)
* 14:18 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1250.eqiad.wmnet with reason: host reimage
* 14:15 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, dreamrimmer: Continuing with sync
* 14:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T371742)', diff saved to https://phabricator.wikimedia.org/P71808 and previous config saved to /var/cache/conftool/dbconfig/20250106-141520-ladsgroup.json
* 14:14 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2042.codfw.wmnet with reason: host reimage
* 14:12 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, dreamrimmer: Backport for gerrit:1107028Add mergehistory to import and transwiki on en.wikibooks (T382785), gerrit:1108171Add suppressredirect and delete-redirect to en.wikinews reviewers (T382887) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 14:12 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1250.eqiad.wmnet with reason: host reimage
* 14:10 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2043.codfw.wmnet with reason: host reimage
* 14:10 marostegui: Deploy schema change on x1 dbmaint codfw T383052
* 14:08 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2042.codfw.wmnet with reason: host reimage
* 14:07 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2043.codfw.wmnet with reason: host reimage
* 14:07 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for gerrit:1107028Add mergehistory to import and transwiki on en.wikibooks (T382785), gerrit:1108171Add suppressredirect and delete-redirect to en.wikinews reviewers (T382887)
* 14:04 marostegui: Deploy schema change on x1 dbmaint eqiad T383052
* 13:51 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1250.eqiad.wmnet with OS bookworm
* 13:49 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2042
* 13:49 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2042
* 13:49 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2043
* 13:49 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2043
* 13:49 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2042.codfw.wmnet with OS bookworm
* 13:49 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2043.codfw.wmnet with OS bookworm
* 13:47 jayme@cumin1002: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P{wikikube-worker[1250-1252].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
* 13:47 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2042-2043].codfw.wmnet
* 13:46 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2042-2043].codfw.wmnet
* 13:40 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2044.codfw.wmnet
* 13:40 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2044.codfw.wmnet
* 13:40 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2045.codfw.wmnet
* 13:40 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2045.codfw.wmnet
* 13:39 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2045.codfw.wmnet with OS bookworm
* 13:34 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2044.codfw.wmnet with OS bookworm
* 13:18 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2045.codfw.wmnet with reason: host reimage
* 13:15 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2044.codfw.wmnet with reason: host reimage
* 13:13 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1243.eqiad.wmnet with OS bookworm
* 13:12 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2045.codfw.wmnet with reason: host reimage
* 13:11 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2044.codfw.wmnet with reason: host reimage
* 12:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2044
* 12:55 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2044
* 12:54 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2045
* 12:54 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2045
* 12:54 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2044.codfw.wmnet with OS bookworm
* 12:54 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2045.codfw.wmnet with OS bookworm
* 12:54 ladsgroup@deploy2002: Finished scap sync-world: Backport for gerrit:1108141ParserCache: Set connect and recieve timeouts (T378076 T373037) (duration: 13m 39s)
* 12:53 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2044-2045].codfw.wmnet
* 12:52 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2044-2045].codfw.wmnet
* 12:51 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2046.codfw.wmnet
* 12:51 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2046.codfw.wmnet
* 12:51 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2048.codfw.wmnet
* 12:51 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2048.codfw.wmnet
* 12:48 ladsgroup@deploy2002: ladsgroup: Continuing with sync
* 12:46 ladsgroup@deploy2002: ladsgroup: Backport for gerrit:1108141ParserCache: Set connect and recieve timeouts (T378076 T373037) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 12:40 ladsgroup@deploy2002: Started scap sync-world: Backport for gerrit:1108141ParserCache: Set connect and recieve timeouts (T378076 T373037)
* 12:37 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1217.eqiad.wmnet with reason: upgrade kernel
* 12:37 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 5:00:00 on db1217.eqiad.wmnet with reason: upgrade kernel
* 12:34 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1193 (T371742)', diff saved to https://phabricator.wikimedia.org/P71807 and previous config saved to /var/cache/conftool/dbconfig/20250106-123416-ladsgroup.json
* 12:34 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1193.eqiad.wmnet with reason: Maintenance
* 12:34 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1193.eqiad.wmnet with reason: Maintenance
* 12:34 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T371742)', diff saved to https://phabricator.wikimedia.org/P71806 and previous config saved to /var/cache/conftool/dbconfig/20250106-123405-ladsgroup.json
* 12:30 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{wikikube-worker[1245-1249].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
* 12:30 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1249.eqiad.wmnet with OS bookworm
* 12:27 Emperor: swift post wikipedia-commons-local-thumb.f8 --read-acl 'mw:thumbor,mw:media,.r:*' --write-acl 'mw:thumbor,mw:media' ms-fe2009 per T383034
* 12:18 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P71805 and previous config saved to /var/cache/conftool/dbconfig/20250106-121858-ladsgroup.json
* 12:14 jayme@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1243.eqiad.wmnet with OS bookworm
* 12:11 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1249.eqiad.wmnet with reason: host reimage
* 12:07 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1249.eqiad.wmnet with reason: host reimage
* 12:04 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es2023.codfw.wmnet
* 12:04 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:04 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2023.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 12:04 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2023.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 12:03 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P71804 and previous config saved to /var/cache/conftool/dbconfig/20250106-120351-ladsgroup.json
* 12:01 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 11:56 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts es2023.codfw.wmnet
* 11:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2046.codfw.wmnet with OS bookworm
* 11:52 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2048.codfw.wmnet with OS bookworm
* 11:48 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T371742)', diff saved to https://phabricator.wikimedia.org/P71803 and previous config saved to /var/cache/conftool/dbconfig/20250106-114844-ladsgroup.json
* 11:47 moritzm: fix /etc/network/interfaces on doc2002 T382610
* 11:47 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1249.eqiad.wmnet with OS bookworm
* 11:45 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1248.eqiad.wmnet with OS bookworm
* 11:35 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2046.codfw.wmnet with reason: host reimage
* 11:32 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2048.codfw.wmnet with reason: host reimage
* 11:28 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2046.codfw.wmnet with reason: host reimage
* 11:26 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2048.codfw.wmnet with reason: host reimage
* 11:24 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1248.eqiad.wmnet with reason: host reimage
* 11:20 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1248.eqiad.wmnet with reason: host reimage
* 11:16 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:16 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:08 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2048
* 11:08 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2046
* 11:08 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2046
* 11:08 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2048
* 11:08 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2046.codfw.wmnet with OS bookworm
* 11:08 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2048.codfw.wmnet with OS bookworm
* 11:07 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2046,2048].codfw.wmnet
* 11:06 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2046,2048].codfw.wmnet
* 11:00 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1248.eqiad.wmnet with OS bookworm
* 10:58 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1247.eqiad.wmnet with OS bookworm
* 10:52 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1243.eqiad.wmnet with OS bookworm
* 10:51 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1242.eqiad.wmnet with OS bookworm
* 10:49 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2050.codfw.wmnet
* 10:49 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2050.codfw.wmnet
* 10:49 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2049.codfw.wmnet
* 10:49 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2049.codfw.wmnet
* 10:39 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1247.eqiad.wmnet with reason: host reimage
* 10:39 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2049.codfw.wmnet with OS bookworm
* 10:36 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2050.codfw.wmnet with OS bookworm
* 10:35 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1247.eqiad.wmnet with reason: host reimage
* 10:32 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1242.eqiad.wmnet with reason: host reimage
* 10:28 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1242.eqiad.wmnet with reason: host reimage
* 10:19 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2049.codfw.wmnet with reason: host reimage
* 10:16 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2050.codfw.wmnet with reason: host reimage
* 10:15 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1247.eqiad.wmnet with OS bookworm
* 10:15 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2049.codfw.wmnet with reason: host reimage
* 10:13 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1246.eqiad.wmnet with OS bookworm
* 10:13 cgoubert@deploy2002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 10:13 cgoubert@deploy2002: helmfile [codfw] START helmfile.d/admin 'apply'.
* 10:13 cgoubert@deploy2002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 10:13 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2050.codfw.wmnet with reason: host reimage
* 10:12 cgoubert@deploy2002: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 10:12 cgoubert@deploy2002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
* 10:11 cgoubert@deploy2002: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
* 10:11 cgoubert@deploy2002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
* 10:10 cgoubert@deploy2002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
* 10:10 cgoubert@deploy2002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 10:09 cgoubert@deploy2002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
* 10:09 cgoubert@deploy2002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:09 cgoubert@deploy2002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
* 10:08 cgoubert@deploy2002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 10:08 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1242.eqiad.wmnet with OS bookworm
* 10:08 cgoubert@deploy2002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 10:07 cgoubert@deploy2002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 10:07 cgoubert@deploy2002: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1167 (T371742)', diff saved to https://phabricator.wikimedia.org/P71801 and previous config saved to /var/cache/conftool/dbconfig/20250106-100706-ladsgroup.json
* 10:07 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 10:06 claime: Deploying admin_ng external services changes on all kubernetes clusters
* 10:06 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
* 10:06 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1167.eqiad.wmnet with reason: Maintenance
* 10:06 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1167.eqiad.wmnet with reason: Maintenance
* 10:06 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1241.eqiad.wmnet with OS bookworm
* 09:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2050
* 09:55 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2050
* 09:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2049
* 09:55 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2049
* 09:55 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2050.codfw.wmnet with OS bookworm
* 09:55 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2049.codfw.wmnet with OS bookworm
* 09:54 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2049-2050].codfw.wmnet
* 09:54 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1246.eqiad.wmnet with reason: host reimage
* 09:53 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2049-2050].codfw.wmnet
* 09:52 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2052.codfw.wmnet
* 09:52 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2052.codfw.wmnet
* 09:52 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2051.codfw.wmnet
* 09:52 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2051.codfw.wmnet
* 09:51 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2052.codfw.wmnet with OS bookworm
* 09:50 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1246.eqiad.wmnet with reason: host reimage
* 09:47 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1241.eqiad.wmnet with reason: host reimage
* 09:46 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2051.codfw.wmnet with OS bookworm
* 09:43 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1241.eqiad.wmnet with reason: host reimage
* 09:41 dcausse: repooling wdqs1012
* 09:31 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2052.codfw.wmnet with reason: host reimage
* 09:29 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1246.eqiad.wmnet with OS bookworm
* 09:27 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2051.codfw.wmnet with reason: host reimage
* 09:26 dcausse: depooling wdqs1012 (high lag, forgot to keep it depooled after restarting blazegraph)
* 09:26 marostegui: Reboot db2160 for kernel upgrade T376905
* 09:25 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2160.codfw.wmnet with reason: upgrade kernel
* 09:25 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 5:00:00 on db2160.codfw.wmnet with reason: upgrade kernel
* 09:25 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1245.eqiad.wmnet with OS bookworm
* 09:24 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2052.codfw.wmnet with reason: host reimage
* 09:23 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2051.codfw.wmnet with reason: host reimage
* 09:23 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1241.eqiad.wmnet with OS bookworm
* 09:21 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1240.eqiad.wmnet with OS bookworm
* 09:06 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2052
* 09:06 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2052
* 09:06 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2051
* 09:06 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2051
* 09:06 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2052.codfw.wmnet with OS bookworm
* 09:06 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2051.codfw.wmnet with OS bookworm
* 09:06 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1245.eqiad.wmnet with reason: host reimage
* 09:05 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2051-2052].codfw.wmnet
* 09:04 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2051-2052].codfw.wmnet
* 09:03 awight: UTC morning deployment finished
* 09:02 awight@deploy2002: Finished scap sync-world: Backport for gerrit:1107557bjnwikiquote: add wordmark (T382777) (duration: 11m 59s)
* 09:02 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1240.eqiad.wmnet with reason: host reimage
* 09:00 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1245.eqiad.wmnet with reason: host reimage
* 08:58 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1240.eqiad.wmnet with reason: host reimage
* 08:57 awight@deploy2002: awight, anzx: Continuing with sync
* 08:55 awight@deploy2002: awight, anzx: Backport for gerrit:1107557bjnwikiquote: add wordmark (T382777) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 08:50 awight@deploy2002: Started scap sync-world: Backport for gerrit:1107557bjnwikiquote: add wordmark (T382777)
* 08:46 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2053.codfw.wmnet
* 08:46 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2053.codfw.wmnet
* 08:46 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2054.codfw.wmnet
* 08:46 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2054.codfw.wmnet
* 08:46 awight@deploy2002: Finished scap sync-world: Backport for gerrit:1106018Change license on ptwikinews, nlwikinews and rowikinews to cc-by-4.0 (T382649) (duration: 09m 28s)
* 08:44 jelto@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker[2053-2054].codfw.wmnet
* 08:44 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2053-2054].codfw.wmnet
* 08:43 jelto@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker[2053-2054].codfw.wmnet
* 08:43 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2053-2054].codfw.wmnet
* 08:42 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2053.codfw.wmnet with OS bookworm
* 08:41 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2054.codfw.wmnet with OS bookworm
* 08:41 awight@deploy2002: dreamrimmer, awight: Continuing with sync
* 08:41 awight@deploy2002: dreamrimmer, awight: Backport for gerrit:1106018Change license on ptwikinews, nlwikinews and rowikinews to cc-by-4.0 (T382649) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 08:39 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1245.eqiad.wmnet with OS bookworm
* 08:38 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1240.eqiad.wmnet with OS bookworm
* 08:37 jayme@cumin1002: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P{wikikube-worker[1245-1249].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
* 08:37 jayme@cumin1002: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P{wikikube-worker[1240-1244].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
* 08:36 awight@deploy2002: Started scap sync-world: Backport for gerrit:1106018Change license on ptwikinews, nlwikinews and rowikinews to cc-by-4.0 (T382649)
* 08:29 awight@deploy2002: awight, dreamrimmer: Backport for gerrit:1106018Change license on ptwikinews, nlwikinews and rowikinews to cc-by-4.0 (T382649) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 08:25 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2053.codfw.wmnet with reason: host reimage
* 08:24 awight@deploy2002: Started scap sync-world: Backport for gerrit:1106018Change license on ptwikinews, nlwikinews and rowikinews to cc-by-4.0 (T382649)
* 08:22 awight@deploy2002: Finished scap sync-world: Backport for [[gerrit:1106940[arwiki] Add templateeditor user group (T382784)]] (duration: 14m 48s)
* 08:22 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2054.codfw.wmnet with reason: host reimage
* 08:18 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2053.codfw.wmnet with reason: host reimage
* 08:18 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2054.codfw.wmnet with reason: host reimage
* 08:18 dcausse: restarting blazegraph on wdqs1012 (stuck with high thread count)
* 08:17 awight@deploy2002: awight, hubaishan: Continuing with sync
* 08:15 dcausse: restarting blazegraph on wdqs1014 (BlazegraphFreeAllocatorsDecreasingRapidly)
* 08:13 awight@deploy2002: awight, hubaishan: Backport for [[gerrit:1106940[arwiki] Add templateeditor user group (T382784)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 08:13 moritzm: installing fastnetmon security updates
* 08:08 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2024 and es2025 T381848', diff saved to https://phabricator.wikimedia.org/P71798 and previous config saved to /var/cache/conftool/dbconfig/20250106-080845-marostegui.json
* 08:07 marostegui@cumin1002: dbctl commit (dc=all): 'Remove es2023 from dbctl and promote es2046 to es5 master T381848 T383026', diff saved to https://phabricator.wikimedia.org/P71797 and previous config saved to /var/cache/conftool/dbconfig/20250106-080755-marostegui.json
* 08:07 awight@deploy2002: Started scap sync-world: Backport for [[gerrit:1106940[arwiki] Add templateeditor user group (T382784)]]
* 08:06 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2024 T381848', diff saved to https://phabricator.wikimedia.org/P71796 and previous config saved to /var/cache/conftool/dbconfig/20250106-080609-marostegui.json
* 08:04 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2023 T383026', diff saved to https://phabricator.wikimedia.org/P71795 and previous config saved to /var/cache/conftool/dbconfig/20250106-080405-marostegui.json
* 08:00 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2054
* 08:00 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2054
* 08:00 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2053
* 08:00 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2053
* 08:00 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2054.codfw.wmnet with OS bookworm
* 08:00 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2053.codfw.wmnet with OS bookworm
* 07:59 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2053-2054].codfw.wmnet
* 07:55 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2053-2054].codfw.wmnet
* 07:29 moritzm: installing systemd bugfix updates from Bookworm point release
* 07:13 marostegui: dbmaint Switchover m3 (phabricator) eqiad master dbproxy1020 -> dbproxy1028 T368874
* 06:54 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es2022.codfw.wmnet
* 06:54 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 06:54 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2022.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 06:54 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2022.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 06:51 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 06:50 marostegui@cumin1002: dbctl commit (dc=all): 'db1193 (re)pooling @ 100%: Repooling after schema change', diff saved to https://phabricator.wikimedia.org/P71794 and previous config saved to /var/cache/conftool/dbconfig/20250106-065050-root.json
* 06:46 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts es2022.codfw.wmnet
* 06:39 marostegui@cumin1002: dbctl commit (dc=all): 'Remove es2022 from dbctl T382946', diff saved to https://phabricator.wikimedia.org/P71792 and previous config saved to /var/cache/conftool/dbconfig/20250106-063940-marostegui.json
* 06:35 marostegui@cumin1002: dbctl commit (dc=all): 'db1193 (re)pooling @ 75%: Repooling after schema change', diff saved to https://phabricator.wikimedia.org/P71791 and previous config saved to /var/cache/conftool/dbconfig/20250106-063545-root.json
* 06:27 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es2020.codfw.wmnet
* 06:27 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 06:27 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 06:27 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 06:23 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 06:20 marostegui@cumin1002: dbctl commit (dc=all): 'db1193 (re)pooling @ 50%: Repooling after schema change', diff saved to https://phabricator.wikimedia.org/P71790 and previous config saved to /var/cache/conftool/dbconfig/20250106-062040-root.json
* 06:19 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts es2020.codfw.wmnet
* 06:18 marostegui@cumin1002: dbctl commit (dc=all): 'Remove es2020 from dbctl T382945', diff saved to https://phabricator.wikimedia.org/P71789 and previous config saved to /var/cache/conftool/dbconfig/20250106-061832-marostegui.json
* 06:08 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es2021.codfw.wmnet
* 06:08 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 06:08 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2021.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 06:08 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2021.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 06:05 marostegui@cumin1002: dbctl commit (dc=all): 'db1193 (re)pooling @ 25%: Repooling after schema change', diff saved to https://phabricator.wikimedia.org/P71787 and previous config saved to /var/cache/conftool/dbconfig/20250106-060534-root.json
* 06:04 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 06:00 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts es2021.codfw.wmnet
* 05:57 marostegui@cumin1002: dbctl commit (dc=all): 'Remove es2021 from dbctl T382944', diff saved to https://phabricator.wikimedia.org/P71786 and previous config saved to /var/cache/conftool/dbconfig/20250106-055726-marostegui.json
* 05:50 marostegui@cumin1002: dbctl commit (dc=all): 'db1193 (re)pooling @ 10%: Repooling after schema change', diff saved to https://phabricator.wikimedia.org/P71785 and previous config saved to /var/cache/conftool/dbconfig/20250106-055029-root.json
* 03:39 tstarling@deploy2002: Finished scap sync-world: Backport for gerrit:1101239Enable canShellboxGetTempUrl everywhere (T292322) (duration: 21m 53s)
* 03:31 tstarling@deploy2002: tstarling: Continuing with sync
* 03:31 tstarling@deploy2002: tstarling: Backport for gerrit:1101239Enable canShellboxGetTempUrl everywhere (T292322) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 03:17 tstarling@deploy2002: Started scap sync-world: Backport for gerrit:1101239Enable canShellboxGetTempUrl everywhere (T292322)
== 2025-01-04 ==
* 20:02 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on doc1003.eqiad.wmnet with reason: dz
* 20:01 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on doc1003.eqiad.wmnet with reason: dz
== 2025-01-03 ==
* 19:09 aokoth@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on doc2002.codfw.wmnet with reason: Disk Change
* 19:09 aokoth@cumin1002: START - Cookbook sre.hosts.downtime for 5 days, 0:00:00 on doc2002.codfw.wmnet with reason: Disk Change
* 18:47 aokoth@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on doc2002.codfw.wmnet with reason: Disk Change
* 18:46 aokoth@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on doc2002.codfw.wmnet with reason: Disk Change
* 17:15 thcipriani@deploy2002: Finished deploy [gerrit/gerrit@44854d4]: remove developer satisfaction survey banner (gerrit1003.wikimedia.org only) (duration: 00m 10s)
* 17:15 thcipriani@deploy2002: Started deploy [gerrit/gerrit@44854d4]: remove developer satisfaction survey banner (gerrit1003.wikimedia.org only)
* 17:06 thcipriani@deploy2002: Finished deploy [gerrit/gerrit@44854d4]: remove developer satisfaction survey banner (gerrit2002.wikimedia.org only) (duration: 00m 08s)
* 17:06 thcipriani@deploy2002: Started deploy [gerrit/gerrit@44854d4]: remove developer satisfaction survey banner (gerrit2002.wikimedia.org only)
* 17:00 isaranto@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 16:55 thcipriani@deploy2002: Finished deploy [gerrit/gerrit@44854d4]: remove developer satisfaction survey banner (gerrit2003.wikimedia.org only) (duration: 00m 08s)
* 16:55 thcipriani@deploy2002: Started deploy [gerrit/gerrit@44854d4]: remove developer satisfaction survey banner (gerrit2003.wikimedia.org only)
* 12:10 moritzm: renewed internal Ganeti certs in eqsin (would have expired in two days) T382873
* 12:01 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2022 T381848', diff saved to https://phabricator.wikimedia.org/P71781 and previous config saved to /var/cache/conftool/dbconfig/20250103-120132-marostegui.json
* 11:12 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2020 T382945', diff saved to https://phabricator.wikimedia.org/P71780 and previous config saved to /var/cache/conftool/dbconfig/20250103-111255-marostegui.json
* 11:05 marostegui: Switchover es4 codfw master to es2043 dbmaint T381848
* 11:04 marostegui@cumin1002: dbctl commit (dc=all): 'Switchover es4 codfw master', diff saved to https://phabricator.wikimedia.org/P71779 and previous config saved to /var/cache/conftool/dbconfig/20250103-110440-marostegui.json
* 10:35 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2021 T381848', diff saved to https://phabricator.wikimedia.org/P71778 and previous config saved to /var/cache/conftool/dbconfig/20250103-103513-marostegui.json
* 10:06 marostegui@cumin1002: dbctl commit (dc=all): 'db2236 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71777 and previous config saved to /var/cache/conftool/dbconfig/20250103-100603-root.json
* 09:50 marostegui@cumin1002: dbctl commit (dc=all): 'db2236 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71776 and previous config saved to /var/cache/conftool/dbconfig/20250103-095057-root.json
* 09:35 marostegui@cumin1002: dbctl commit (dc=all): 'db2236 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71774 and previous config saved to /var/cache/conftool/dbconfig/20250103-093552-root.json
* 09:20 marostegui@cumin1002: dbctl commit (dc=all): 'db2236 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71773 and previous config saved to /var/cache/conftool/dbconfig/20250103-092046-root.json
* 09:05 marostegui@cumin1002: dbctl commit (dc=all): 'db2236 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71772 and previous config saved to /var/cache/conftool/dbconfig/20250103-090541-root.json
* 09:03 marostegui: Upgrade db2236 to 10.11.10 s4 codfw dbmaint T378940
* 09:02 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2236.codfw.wmnet with reason: upgrade
* 09:02 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on db2236.codfw.wmnet with reason: upgrade
* 09:02 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2236 to upgrade to 10.11.10 T378940', diff saved to https://phabricator.wikimedia.org/P71771 and previous config saved to /var/cache/conftool/dbconfig/20250103-090215-marostegui.json
* 08:35 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2115.codfw.wmnet
* 08:35 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:35 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2115.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 08:33 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2115.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 08:29 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 08:24 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2115.codfw.wmnet
* 07:36 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on db1193.eqiad.wmnet with reason: maintenance
* 07:36 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on db1193.eqiad.wmnet with reason: maintenance
* 07:23 marostegui@cumin1002: dbctl commit (dc=all): 'Remove db2115 from dbctl T362949', diff saved to https://phabricator.wikimedia.org/P71770 and previous config saved to /var/cache/conftool/dbconfig/20250103-072349-marostegui.json
* 06:32 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2116.codfw.wmnet
* 06:32 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 06:32 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2116.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 06:32 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2116.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 06:28 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 06:23 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2116.codfw.wmnet
== 2025-01-02 ==
* 22:23 wfan: SmashPig upgraded from to 
* 21:26 urbanecm@deploy2002: Finished scap sync-world: Backport for [[gerrit:1106316[Growth] Remove Marketing campaign (T382499)]], gerrit:1107561gomwiki: Use wikitext talk pages by default (T382810) (duration: 21m 42s)
* 21:18 urbanecm@deploy2002: urbanecm: Continuing with sync
* 21:17 urbanecm@deploy2002: urbanecm: Backport for [[gerrit:1106316[Growth] Remove Marketing campaign (T382499)]], gerrit:1107561gomwiki: Use wikitext talk pages by default (T382810) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
* 21:04 urbanecm@deploy2002: Started scap sync-world: Backport for [[gerrit:1106316[Growth] Remove Marketing campaign (T382499)]], gerrit:1107561gomwiki: Use wikitext talk pages by default (T382810)
* 18:23 bd808@deploy2002: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 18:23 bd808@deploy2002: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 18:23 bd808@deploy2002: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 18:22 bd808@deploy2002: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 18:22 bd808@deploy2002: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 18:22 bd808@deploy2002: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 18:22 bd808@deploy2002: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 18:21 bd808@deploy2002: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 14:28 marostegui@cumin1002: dbctl commit (dc=all): 'Remove db2116 from dbctl T362950', diff saved to https://phabricator.wikimedia.org/P71768 and previous config saved to /var/cache/conftool/dbconfig/20250102-142806-marostegui.json
* 11:49 mvernon@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ms-be1066.eqiad.wmnet
* 11:49 mvernon@cumin1002: START - Cookbook sre.hosts.remove-downtime for ms-be1066.eqiad.wmnet
* 11:31 marostegui: dbmaint s8 db1193 eqiad rebuild pagelinks and recentchanges and deploy schema change on revision table T367856 T382842
* 11:29 mvernon@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ms-be1066.eqiad.wmnet with reason: vacuum three container dbs
* 11:28 mvernon@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on ms-be1066.eqiad.wmnet with reason: vacuum three container dbs
* 11:22 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on db1193.eqiad.wmnet with reason: maintenance
* 11:22 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 5 days, 0:00:00 on db1193.eqiad.wmnet with reason: maintenance
* 11:11 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1193 T381993', diff saved to https://phabricator.wikimedia.org/P71766 and previous config saved to /var/cache/conftool/dbconfig/20250102-111105-marostegui.json
* 11:09 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1209 to s8 primary T381993', diff saved to https://phabricator.wikimedia.org/P71765 and previous config saved to /var/cache/conftool/dbconfig/20250102-110923-marostegui.json
* 11:09 marostegui: Starting s8 eqiad failover from db1193 to db1209 - T381993
* 11:03 marostegui@cumin1002: dbctl commit (dc=all): 'Remove db1209 from API/vslow/dump T381993', diff saved to https://phabricator.wikimedia.org/P71764 and previous config saved to /var/cache/conftool/dbconfig/20250102-110305-root.json
* 11:02 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1209 with weight 0 T381993', diff saved to https://phabricator.wikimedia.org/P71763 and previous config saved to /var/cache/conftool/dbconfig/20250102-110232-root.json
* 11:00 root@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s8 T381993
* 11:00 root@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 32 hosts with reason: Primary switchover s8 T381993
* 10:56 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dbproxy2004.codfw.wmnet
* 10:56 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:56 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy2004.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 10:56 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy2004.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 10:56 marostegui: dbmaint codfw Decommissioned dbproxy200[1-4] m1 m2 m3 m5 T381962
* 10:56 marostegui: dbmaint codfw Decommissioned dbproxy200[1-4] T381962
* 10:52 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 10:48 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts dbproxy2004.codfw.wmnet
* 10:14 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dbproxy2003.codfw.wmnet
* 10:14 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:14 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 10:14 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 10:10 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 10:06 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts dbproxy2003.codfw.wmnet
* 09:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 100%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71762 and previous config saved to /var/cache/conftool/dbconfig/20250102-095838-root.json
* 09:56 Emperor: restart swift-object on ms-be1075
* 09:43 marostegui@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 75%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71761 and previous config saved to /var/cache/conftool/dbconfig/20250102-094332-root.json
* 09:28 marostegui@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 50%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71760 and previous config saved to /var/cache/conftool/dbconfig/20250102-092827-root.json
* 09:13 marostegui@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 25%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71759 and previous config saved to /var/cache/conftool/dbconfig/20250102-091322-root.json
* 09:01 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dbproxy2002.codfw.wmnet
* 09:01 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 09:01 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 09:01 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 08:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 10%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71758 and previous config saved to /var/cache/conftool/dbconfig/20250102-085816-root.json
* 08:57 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 08:53 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts dbproxy2002.codfw.wmnet
* 08:43 marostegui@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 5%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71757 and previous config saved to /var/cache/conftool/dbconfig/20250102-084312-root.json
* 08:28 marostegui@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 1%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71756 and previous config saved to /var/cache/conftool/dbconfig/20250102-082806-root.json
* 08:17 marostegui@cumin1002: dbctl commit (dc=all): 'db2171 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71755 and previous config saved to /var/cache/conftool/dbconfig/20250102-081722-root.json
* 08:08 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dbproxy2001.codfw.wmnet
* 08:08 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:08 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 08:08 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
* 08:04 marostegui@cumin1002: START - Cookbook sre.dns.netbox
* 08:02 marostegui@cumin1002: dbctl commit (dc=all): 'db2171 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71754 and previous config saved to /var/cache/conftool/dbconfig/20250102-080216-root.json
* 08:00 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts dbproxy2001.codfw.wmnet
* 07:47 marostegui@cumin1002: dbctl commit (dc=all): 'db2171 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71753 and previous config saved to /var/cache/conftool/dbconfig/20250102-074711-root.json
* 07:35 marostegui: Stop haproxy on dbproxy200[14] T381962
* 07:34 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on dbproxy2004.codfw.wmnet with reason: maintenance
* 07:34 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on dbproxy2004.codfw.wmnet with reason: maintenance
* 07:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on dbproxy2003.codfw.wmnet with reason: maintenance
* 07:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on dbproxy2003.codfw.wmnet with reason: maintenance
* 07:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on dbproxy2002.codfw.wmnet with reason: maintenance
* 07:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on dbproxy2002.codfw.wmnet with reason: maintenance
* 07:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on dbproxy2001.codfw.wmnet with reason: maintenance
* 07:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on dbproxy2001.codfw.wmnet with reason: maintenance
* 07:32 marostegui@cumin1002: dbctl commit (dc=all): 'db2171 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71752 and previous config saved to /var/cache/conftool/dbconfig/20250102-073206-root.json
* 07:17 marostegui@cumin1002: dbctl commit (dc=all): 'db2171 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71751 and previous config saved to /var/cache/conftool/dbconfig/20250102-071700-root.json
* 07:13 root@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2171.codfw.wmnet onto db2123.codfw.wmnet
* 06:37 root@cumin1002: START - Cookbook sre.mysql.clone of db2171.codfw.wmnet onto db2123.codfw.wmnet
* 06:32 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2171.codfw.wmnet with reason: maintenance
* 06:32 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2171.codfw.wmnet with reason: maintenance
* 06:32 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2171 to clone db2123 T382744', diff saved to https://phabricator.wikimedia.org/P71750 and previous config saved to /var/cache/conftool/dbconfig/20250102-063218-marostegui.json
* 06:29 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2123.codfw.wmnet with reason: maintenance
* 06:29 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2123.codfw.wmnet with reason: maintenance
* 00:03 sukhe@cumin1002: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool site eqsin for service: ncredir-addrs [reason: no reason specified, no task ID specified]
* 00:03 sukhe@cumin1002: START - Cookbook sre.dns.admin DNS admin: depool site eqsin for service: ncredir-addrs [reason: no reason specified, no task ID specified]
==Other archives==
Category:SAL
Category:Operations