Change:
Any test that runs Gracefulshutdown seems to be having issues. Reference: In this case, they are failing consistently. The tests were flaky but as of 9/29, they have been failing ever since. and I were looking into these failures. We are able to reproduce on a GCP instance but we are seeing failures to sending dbus shutdown signals. We have looked into this code but it does not look like any code is responsible for this failure. One thing we did see is that there was a new release of COS around then. We are seeing failures in the logs. Running this test locally we can sometimes see issues with dbus. Running this locally (after copying the image config): /sig node
/cc
/cc
/triage accepted /priority important-soon
Looking in the logs of the last successful job: And viewing the kernel logs, I don't see any errors related to dbus. Last successful job linux version: The next job fails and the main difference seems to be a linux kernel update.
/cc
/cc
I was able to repro this on COS M109 as follows, while the below works fine on COS M105. Will dig a bit deeper if this is an issue in COS or systemd upstream regression.
thank you for looking into this! (adding who authored the Dbus restart code) What we are seeing here would be something other than a regression. This problem affects Google's COS, Fedora CoreOS (and its RHEL-based equivalent), Fedora Workstations, and most likely any other Linux distribution, whether container optimised or not, that is not Debian or based on Debian, such as Ubuntu. Both Debian and Ubuntu are unaffected as they handle Dbus restarts slightly differently. TL;DR Restarting the Dbus broker service at runtime is generally not recommended or supported. Many systemd components (but also other software and services) that rely on Dbus wouldn't reconnect to a new Unix socket, which would have the same path (location) as before, following the Dbus broker restart. This can lead to anything from authentication timeouts (since the systemd-logind and the polkitd services would be impacted) to the system generally broken following Dbus broker restart - this is especially the case for desktop (such as Fedora Workstation) systems where there are a lot of variety of services that rely on working Dbus connectivity. Different software might handle this differently and choose to reconnect on a socket write or read timeout. However, systemd at large is not one of these. Lennart has confirmed this on a number of occasions per (selected from a sample of issues): Debian and its derivatives, such as Ubuntu or Linux Mint, etc., are not affected as they opt to trigger the restart (or activation, if you wish) of key system services (such as the systemd-logind) following a Dbus broker restart. I recommend removing the Dbus broker restart from the test code base to fix the failing tests. If there were no objections, then I would be happy to do this work. Additionally, we could see about porting Debian's behaviour to Google's COS and Fedora CoreOS (and, as such, the Red Hat RHEL-based equivalent) around restarting systemd-logind on Dbus restart. But that would be a bonus work and outside of this issue's scope. Typically on a Fedora CoreOS distribution, the default list of Dbus connections just after system startup might look as follows: <details<summaryDbus connections list on Fedora CoreOS after startup</summary</detailsSeveral key systemd services are connected, such as the systemd-logind and the systemd-homed, plus the polkitd service. Restarting Dbus would reduce this list to the following: <details<summaryDbus connections list on Fedora CoreOS after Dbus broker restart</summary</detailsThere is almost nothing connected following the restart. Any attempt to introspect the /org/freedesktop/login1 object would now fail per: <details<summaryObject introspection run</summary</detailsRestarting the systemd-logind service causes a fresh process to reconnect to the Dbus bus per: <details<summaryDbus connections list on Fedora CoreOS after systemd-logind restart</summary</detailsThis resolves the problem of accessing the /org/freedesktop/login1 object per: <details<summaryObject introspection run</summary</detailsHowever, other services might still be affected and would need to be restarted. I mentioned that Debian and its derivatives are not affected as they trigger the restart of key systemd services, especially the systemd-logind service. This can be seen in the following output - pay close attention to the PID of the systemd-logind service on the list (using Debian 12 "bookworm" as an example): <details<summaryDbus connections list on Debian 12 following Dbus broker restart</summary</detailsNote that for this particular Debian 12 system, the initial state of the connections was different. The following is the state immediately after the system starts up, this is a desktop flavour with a lot of other software that relies on Dbus, and that will also fail to reconnect correctly per: <details<summaryDbus connections list on Debian 12 after startup</summary</detailsWhen the Dbus broker is restarted, and a service that exposes an object hasn't reconnected to the new Unix socket, the following breakdown in connection between the client, the Dbus bus, and the service of interest in question can be seen. Any attempt to access the /org/freedesktop/login1 object will not work: <details<summaryObject introspection output</summary</detailsThis failure can also be seen on a bit lower level too, per: <details<summaryThe busctl strace output</summary</detailsA test has been in the past to catch issues following the Dbus broker service restart as part of a genuine issue that consisted of adding support for reconnecting to the Dbus socket in case of errors or connection failure. This test would work fine on Debian and Ubuntu, as these Linux distributions aren't affected as much as we already learned. Specific commit: This was in response to a reported issue: Given that tests are nowadays commonly run on either Googe Container-optimised OS (COS) or Fedora CoreOS, the addition of the Dbus broker restart would have side effects: Any use of would either appear as hang or take a long time to time out Attempt to sign into a given compute instance using SSH would have taken a long time or appear as a hang connection Dispatching requests to Dbus to send a signal to some other participating services can also take a long time before eventually timing out Some combination of the above might help explain why a lot of our tests were timing out either on connection (SSH) or execution (Ginko, etc.). Related code: The code responsible for sending a signal (SIGHUP) to systemd-logind to ask it to reload its configuration: The above is an interesting take on sending the SIGHUP signal to a process since we don't need to find out what PID the process in question has and just ask Dbus to do the work for us - a bit unorthodox and relies on the service that when breaks would render other interactions with the system and other services unreliable. Related upstream issues and Pull Requests for reference: Linux distributions used for testing: <details<summaryA list of Linux distributions showing the /etc/os-release file content</summary</details
I took the time to diagnose it this week, and I tried to find a way around it. The previous purpose was to fix that kubelet could not work after dbus restart. Currently, this issue has been fixed in kubelet for a long time, and I have no opinion on deleting this test.
Hi This is working without issues. The issue is not between kubelet and Dbus, but between Dbus and other services that do not handle the Dbus broker being restarted. We are good on the kubelet front. :smile:

Discussion:
return nil }, nodeStatusUpdateTimeout, pollInterval).Should(gomega.Succeed()) }) <del> ginkgo.It("after restart dbus, should be able to gracefully shutdown", func(ctx context.Context) { // allows manual restart of dbus to work in Ubuntu. err := overlayDbusConfig() framework.ExpectNoError(err) defer func() { err := restoreDbusConfig() framework.ExpectNoError(err) }() ginkgo.By("Restart Dbus") err = restartDbus() framework.ExpectNoError(err) gomega.Eventually(ctx, func(ctx context.Context) error { // re-send the shutdown signal in case the dbus restart is not done ginkgo.By("Emitting Shutdown signal") err = emitSignalPrepareForShutdown(true) if err != nil { return err } isReady := getNodeReadyStatus(ctx, f) if isReady { return fmt.Errorf("node did not become shutdown as expected") } return nil }, nodeStatusUpdateTimeout, pollInterval).Should(gomega.Succeed()) }) </del> }) ginkgo.Context("when gracefully shutting down with Pod priority", func() {

return isNodeReady(&nodeList.Items[0]) } <del> func restartDbus() error { cmd := "systemctl restart dbus" _, err := runCommand("sh", "-c", cmd) return err } </del> func systemctlDaemonReload() error { cmd := "systemctl daemon-reload" _, err := runCommand("sh", "-c", cmd)