Change:
What happened: The node lease feature becomes beta and enables by default in v1.14. After upgrading our cluster to v1.14, we found some nodes become not ready occasionally and later become ready again after a few seconds. In order to have a quick failure detection and recovery, our kubelet has a flag setting and controller-manager has another flag setting . If we use node lease feature, the kubelet doesn't report it's status as frequently as specified by flag , and use node lease mechanism to report that node is alive. the default interval at which node lease renew is hardcoded 10s, but it can't be exactly 10s and may be a little bit more than 10s because of some network latency. when controller-manager doesn't receive messages from kubelet in every 10s in our case, it will mark this node as unreachable and not ready state, but in the next reconcile cycle, it receives the lease update message and mark this node as reachable again, so the node status is flapping between ready and not ready state. This seems a breaking change that setting kubelet's lease renewing interval to a hardcoded 10s, even though our node status update frequency has a little bit small value, node lease should align kubelet's previous behavior and renew as frequently as to reporting alive message, otherwise controller-manager will mark this node as unreachable. What you expected to happen: So we should not merely set node lease renewing interval to 10s, maybe depending on the has a more heuristic value. How to reproduce it (as minimally and precisely as possible): a v1.14 kubernetes cluster set flag to kubelet and to controller-manager. wait a few minute, 'll find some log regarding in controller-manager's log. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig node /kind bug /assign
smells like a regression WDYT?
cc
- not a regression - I think the only issue is that we may be missing some documentation, but other than that I think everything looks fine. So how it works now is: There are two fields in Kubelet Config: (a) NodeStatusUpdateFrequency: (b) NodeStatusReportFrequency The comments there are pretty detailed and you should read them, but in short the first one roughly translates to how frequenty the lease object is send and the second how frequenty node status is updated (assuming nothing changed in the meantime). IIRC, there was a decision to not expose it via a flag, because flags in kubelet are deprecated and we're moving to config. So you can actually mitigate the problem by setting the second field in the config I mentioned above to some smaller value (like 5s) and then you will get Node Status updates every 5s as you want. So it's possible to achieve what you want without any code changes. That said, enabling for changing the default lease duration to something larger than 10s, sounds reasonable.
How about making the default lease duration 12s (or 14s) ? I can send out a PR once a new default is agreed upon.
thanks for comments. But what if we both want node lease feature enables (as this is an indeed helpful feature) and heartbeat with apiserver within some smaller interval (like 5s) via node lease mechanism?
IMHO, we should add a filed in Kubelet Config to let user tune the node lease renewing interval, WDYT?
That sounds like the cleanest option to me (rather than some playing with existing params). - who is the owner of the feature on Node side.

Discussion:
klet.softAdmitHandlers.AddPodAdmitHandler(lifecycle.NewNoNewPrivsAdmitHandler(klet.containerRuntime)) if utilfeature.DefaultFeatureGate.Enabled(features.NodeLease) { <del> klet.nodeLeaseController = nodelease.NewController(klet.clock, klet.heartbeatClient, string(klet.nodeName), kubeCfg.NodeLeaseDurationSeconds, klet.onRepeatedHeartbeatFailure) </del> <ins> klet.nodeLeaseController = nodelease.NewController(klet.clock, klet.heartbeatClient, string(klet.nodeName), kubeCfg.NodeLeaseDurationSeconds, kubeCfg.NodeStatusUpdateFrequency.Duration, klet.onRepeatedHeartbeatFailure) </ins> } klet.softAdmitHandlers.AddPodAdmitHandler(lifecycle.NewProcMountAdmitHandler(klet.containerRuntime))

) const ( <del> // renewInterval is the interval at which the lease is renewed renewInterval = 10 * time.Second </del> <ins> // defaultRenewInterval is the default interval at which the lease is renewed defaultRenewInterval = 10 * time.Second </ins> // maxUpdateRetries is the number of immediate, successive retries the Kubelet will attempt // when renewing the lease before it waits for the renewal interval before trying again, // similar to what we do for node status retries

} // NewController constructs and returns a controller <del> func NewController(clock clock.Clock, client clientset.Interface, holderIdentity string, leaseDurationSeconds int32, onRepeatedHeartbeatFailure func()) Controller { </del> <ins> func NewController(clock clock.Clock, client clientset.Interface, holderIdentity string, leaseDurationSeconds int32, nodeStatusUpdateFrequency time.Duration, onRepeatedHeartbeatFailure func()) Controller { </ins> var leaseClient coordclientset.LeaseInterface if client != nil { leaseClient = client.CoordinationV1().Leases(corev1.NamespaceNodeLease) } <ins> renewInterval := defaultRenewInterval // Users are able to decrease the timeout after which nodes are being // marked as "Ready: Unknown" by NodeLifecycleController to values // smaller than defaultRenewInterval. Until the knob to configure // lease renew interval is exposed to user, we temporarily decrease // renewInterval based on the NodeStatusUpdateFrequency. if renewInterval > nodeStatusUpdateFrequency { renewInterval = nodeStatusUpdateFrequency } </ins> return &controller{ client:                     client, leaseClient:                leaseClient,