Change:
/priority critical-urgent /priority failing-test /kind bug /status approved-for-milestone /area platform/gke owns the test This test has been failing since at least 2017-11-30 for the following job: https://k8s- This job is on the , and prevents us from cutting v1.9.0 (kubernetes/sig-release). Is there work ongoing to bring this test back to green? /assign Pulling this out of into its own issue. It might be GKE specific? But the GCE downgrade jobs are failing right now, so I don't have enough data to say for sure
Copying latest findings here
Agree, seems like a problem with addon manager /cc
At this point it seems somehow vague to tell which exact component is causing the issue. Probably worth to solve the fundamental downgrade issue () first.
Digging in a bit ... Looks like the test installs a Log Provider using this: and fails here as it finds 2 agents instead of just the one that it installs: Probably because i am guessing fluentd was switched on in this environment? This seems to have happened around 11/30, ring any bells to anyone?
Test doesn't install anything, there's a valid error in addon manager, which results in two addon DaemonSets running after the downgrade Flunetd is there since forever, as DaemonSet for 3-4 releases already, nothing has changed in it, except for the version, which should be handled adequately by addon manager, but it's not
Ack thanks!
Another clue a better one i hope, grepping through the UUID(s) for the fluentd containers - spotted the following and trying to look for why it failed, saw the following These logs are from:
/sig network
This log line usually shows up when kubelet restarts. It should recover after a while.
Can someone from sig-node investigate why kubelet was restarted and if it's normal?
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
will take a look at kubelet restarts to see if that's normal
I think I got the root cause. Starting from addon-manager's log of the current run (https://k8s-): So clearly something changed for ClusterRoleBinding "kubelet-cluster-admin" between 1.8 and 1.9, upon the downgrade, addonmanager tried to apply the change to an immutable field and failed. Due to some implementation details in , when any error occurred during create/update, the prune operation will not be performed. Hence fluentd-gcp-v2.0.9 daemonset was created but fluentd-gcp-v2.0.10 daemonset was not pruned. Also due to some implementation details in addonmanager, fluentd daemonset and fluentd confimap are managed in seperate label groups, which explains why fluentd-gcp-config-v1.2.3 was pruned successfully. might know about the ClusterRoleBinding "kubelet-cluster-admin" change?
cc who last touched these files:
Seems like ClusterRoleBinding kubelet-cluster-admin does change a bit in the legacy-kubelet-user-disable case:
It seems related to In this case, we create a new cluster, which gets the new binding. Then we try to downgrade, which ends up attempting to switch back to the old binding. I wonder if we can cherry-pick something to 1.8 to make it keep the new binding upon downgrade?
cc and who reviewed the above PR.
From addonmanager perspective, one way to achieve this is by changing the Reconcile mode to EnsureExist mode.
8 created kubelet-cluster-admin binding to system:node role in ReconcileMode If we want 1.9 to tolerate that and not remove it, we should leave the kubelet-cluster-admin binding to system:node role with no subjects and EnsureExists mode Separately in 1.9 we can create a new kubelet-bootstrapper binding to system:node-bootstrapper for the kubelet subject with reconcile mode
(Which is what it looks like did… the later change of what role it bound to was the breaking change)
should have left the existing binding alone and created a new binding for the new role
Just to make sure we're on the same page: We didn't see any problem in 1.8-1.9 upgrade tests. This only shows up in the downgrade test where we start with a fresh 1.9 cluster and downgrade to 1.8. It sounds like your plan will help with both directions, but just want to be clear. Can you work on a PR or do I need to find someone?
Nice work So we don't really capture the addon manager logs do we? Looks like you had to pick one up from the running system, is that right?
Yeah, unfortunately addon manager logs on GKE CIs is not publicly visible. There may be ways to retrieve the same logs for history runs, just that getting it from a running cluster seems easiest to me.
Quickly looked at Kubelet and netconfig is not ready error message should be red herring. Right after that, all nodes are ready with the following logging: Will look more.
Ahh, after refreshing the issue, and found that already identified the root cause. Nice work and thanks!
The add-on manager would still have hit issues trying to reapply the changed rolebinding, but the 1.8 binding granted a superset of permissions, so we likely didn't notice. Correct opened
Thanks at least my investigation triggered a deeper look which seems to have helped the cause. yay!
Reopening until we confirm the fix worked: https://k8s-
Looks like we are green - https://k8s-

Discussion:
roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole <ins> name: system:node subjects: [] --- # This is required so that new clusters still have bootstrap permissions apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kubelet-bootstrap labels: kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole </ins> name: system:node-bootstrapper subjects: - apiGroup: rbac.authorization.k8s.io