Repository: siderolabs/talos

Issue #9980: Talos cannot read some Kubernetes APIs after K8s v1.32.0
Created: 2024-12-17T17:05:25Z
Bug Report
Talos cannot read some Kubernetes APIs after K8s v1.32.0.
Logs
[talos] kubernetes registry node watch error {"component": "controller-runtime", "controller": "cluster.KubernetesPullController", "error": "failed to list *v1.Node: nodes is forbidden: User \"system:node:m0\" cannot list resource \"nodes\" in API group \"\" at the cluster scope: node 'm0' cannot read all nodes, only its own Node object"}

Environment

Talos version: v1.9.0
Kubernetes version: v1.32.0
Platform: metal

Duplicate issue #9991: CP shows Ready: false after reboot
Created: 2024-12-19T08:44:47Z
Bug Report
Description
I updated my Nodes to Talos 1.9.0 and Kubernetes 1.32.0. After rebooting a CP with talosctl the task gets stuck on:
◳ watching nodes: [rpi03]
    * rpi03: stage: RUNNING ready: false unmetCond: [name:"nodeReady" reason:"node \"rpi03\" status is not available yet"]
talosctl dashboard rpi03 also shows Ready: false while everything is working fine.
a kubectl get nodes shows the nodes as Status: Ready
Logs
k describe node rpi03
Events:
  Type     Reason                   Age                    From             Message
  ----     ------                   ----                   ----             -------
  Normal   Starting                 6m9s                   kube-proxy
  Normal   RegisteredNode           45m                    node-controller  Node rpi03 event: Registered Node rpi03 in Controller
  Normal   RegisteredNode           21m                    node-controller  Node rpi03 event: Registered Node rpi03 in Controller
  Normal   RegisteredNode           21m                    node-controller  Node rpi03 event: Registered Node rpi03 in Controller
  Normal   Shutdown                 9m7s                   kubelet          Shutdown manager detected shutdown event
  Normal   NodeNotReady             9m7s                   kubelet          Node rpi03 status is now: NodeNotReady
  Warning  InvalidDiskCapacity      6m24s                  kubelet          invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  6m24s                  kubelet          Updated Node Allocatable limit across pods
  Normal   Starting                 6m24s                  kubelet          Starting kubelet.
  Warning  Rebooted                 6m23s (x3 over 6m24s)  kubelet          Node rpi03 has been rebooted, boot id: 295c8af1-a7eb-4b9e-b308-d5dab0fbe475
  Normal   NodeHasSufficientMemory  6m23s (x4 over 6m24s)  kubelet          Node rpi03 status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    6m23s (x4 over 6m24s)  kubelet          Node rpi03 status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     6m23s (x4 over 6m24s)  kubelet          Node rpi03 status is now: NodeHasSufficientPID
  Normal   NodeNotReady             6m23s (x3 over 6m24s)  kubelet          Node rpi03 status is now: NodeNotReady
  Normal   NodeReady                6m23s                  kubelet          Node rpi03 status is now: NodeReady

Environment

Talos version:

Client:
        Tag:         v1.9.0
        SHA:         3cb25ceb
        Built:
        Go version:  go1.23.4
        OS/Arch:     darwin/arm64
Server:
        NODE:        rpi04
        Tag:         v1.9.0
        SHA:         3cb25ceb
        Built:
        Go version:  go1.23.4
        OS/Arch:     linux/arm64
        Enabled:     RBAC


Kubernetes version:

Client Version: v1.32.0
Kustomize Version: v5.5.0
Server Version: v1.32.0


Platform: RasPi 4

Closing comment (2025-01-17T15:05:53Z): Hm, it actually looks like this is related to #9980 (comment) as this actually started happening once I upgraded to K8s 1.32... and my nodes become ready in the dashboard immediately once I change that flag. However I do have the cluster.discovery.registries.kubernetes.disabled setting to true, and just cluster.discovery.enabled to true.